Oct 30 23:57:53.669052 ip-10-0-140-246 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:57:54.092758 ip-10-0-140-246 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:57:54.092758 ip-10-0-140-246 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:57:54.092758 ip-10-0-140-246 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:57:54.092758 ip-10-0-140-246 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:57:54.099119 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099044 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.099119 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099118 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099122 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099126 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099128 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099131 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099134 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099136 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099140 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099143 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099145 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099148 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099151 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099155 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099159 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099162 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099164 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099168 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099172 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099175 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.099185 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099177 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099180 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099182 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099185 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099193 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099196 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099199 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099202 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099205 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099207 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099210 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099212 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099215 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099220 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099222 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099225 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099227 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099230 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099232 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.099630 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099235 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099237 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099240 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099242 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099245 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099247 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099250 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099253 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099255 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099258 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099260 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099262 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099265 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099270 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099273 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099275 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099278 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099281 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099283 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.100113 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099286 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099289 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099292 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099294 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099297 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099299 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099301 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099304 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099306 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099309 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099312 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099314 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099316 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099319 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099322 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099324 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099327 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099330 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099332 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099335 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.100599 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099337 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099339 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099344 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099347 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099350 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099354 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099357 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099359 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099362 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099365 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099367 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099370 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099373 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099375 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099377 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099380 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099382 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099385 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.101108 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099387 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099390 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099392 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099395 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099397 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099400 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.099402 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100152 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100158 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100161 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100165 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100167 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100170 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100173 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100177 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100182 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100185 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100189 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100193 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.101554 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100196 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100200 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100203 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100205 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100208 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100212 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100214 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100217 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100220 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100222 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100225 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100228 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100230 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100233 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100235 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100238 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100240 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100243 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100245 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100248 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.102061 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100250 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100253 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100255 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100257 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100260 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100263 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100265 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100268 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100270 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100273 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100275 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100278 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100281 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100283 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100286 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100288 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100291 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100294 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100297 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100299 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.102530 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100302 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100304 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100307 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100310 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100312 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100315 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100317 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100320 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100322 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100324 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100327 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100329 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100332 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100334 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100337 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100340 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100343 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100347 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100350 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.103065 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100353 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100357 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100360 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100363 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100365 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100368 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100371 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100373 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100377 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100380 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100382 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100384 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100387 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100390 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100392 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100397 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100400 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100402 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100405 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.103520 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100409 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100412 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100414 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100417 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100419 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100421 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100424 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100426 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100429 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100431 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100433 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100436 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100439 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100441 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100443 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100934 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100939 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100942 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100945 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.103976 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100948 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100950 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100953 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100956 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100958 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100962 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100965 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100967 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100972 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100975 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100977 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100979 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100982 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100985 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100987 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100990 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100992 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100994 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100997 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.104434 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.100999 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101004 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101007 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101009 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101012 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101014 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101016 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101019 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101022 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101024 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101027 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101030 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101032 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101035 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101037 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101040 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101042 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101045 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101048 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.104908 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101051 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101055 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101057 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101060 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101062 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101065 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101067 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101070 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101072 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101075 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101077 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101080 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101082 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101085 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101087 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101090 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101092 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101094 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101097 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101100 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.105468 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101102 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101104 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101107 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101113 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101117 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101121 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101125 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101127 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101130 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101133 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101135 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101138 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101141 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101144 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101146 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101149 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101151 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101154 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.105968 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101156 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101161 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101163 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101166 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101168 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101171 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101173 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101175 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101178 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101180 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101183 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101185 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101187 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101190 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101192 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101197 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101201 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101205 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.106412 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101209 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101212 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101216 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101219 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.101222 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102290 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102304 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102314 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102318 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102323 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102326 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102330 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102334 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102337 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102340 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102344 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102347 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102350 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102353 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102356 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102359 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102362 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102365 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:57:54.106885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102368 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102371 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102374 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102377 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102381 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102384 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102389 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102392 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102395 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102398 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102401 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102404 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102407 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102410 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102414 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102417 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102420 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102424 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102427 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102430 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102434 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102437 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102440 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102444 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:57:54.107396 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102447 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102451 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102454 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102457 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102460 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102463 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102466 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102469 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102472 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102475 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102477 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102480 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102484 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102487 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102491 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102494 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102497 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102500 2578 flags.go:64] FLAG: --help="false" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102503 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102506 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102509 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102512 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102515 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:57:54.108021 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102519 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102522 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102525 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102528 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102531 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102534 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102537 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102540 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102543 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102546 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102549 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102551 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102554 2578 flags.go:64] FLAG: --lock-file="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102557 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102560 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102562 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102568 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102570 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102574 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102586 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102589 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102592 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102595 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102599 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102605 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:57:54.108559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102608 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102611 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102614 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102617 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102620 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102622 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102626 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102629 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102632 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102640 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102643 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102646 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102649 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102652 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102654 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102657 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102660 2578 flags.go:64] FLAG: --port="10250" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102663 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102666 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02422223eb39a111c" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102669 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102671 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102675 2578 flags.go:64] FLAG: --register-node="true" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102677 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102681 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102684 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102687 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:57:54.109162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102689 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102693 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102696 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102699 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102703 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102706 2578 flags.go:64] FLAG: --runonce="false" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102709 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102712 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102714 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102717 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102720 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102723 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102727 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102730 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102733 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102736 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102739 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102742 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102745 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102748 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102751 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102756 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102758 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102761 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102764 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:57:54.110128 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102767 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102770 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102772 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102775 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102778 2578 flags.go:64] FLAG: --v="2" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102796 2578 flags.go:64] FLAG: --version="false" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102800 2578 flags.go:64] FLAG: --vmodule="" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102803 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.102806 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102934 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102937 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102942 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102945 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102948 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102950 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102953 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102955 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102958 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102960 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102964 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102967 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.110919 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102969 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102972 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102974 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102977 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102979 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102982 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102985 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102988 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102992 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102995 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.102997 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103000 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103002 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103005 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103007 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103010 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103012 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103015 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103017 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.111431 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103020 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103022 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103025 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103029 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103031 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103034 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103036 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103039 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103042 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103044 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103047 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103052 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103056 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103059 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103062 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103064 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103067 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103069 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103072 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.111890 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103079 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103082 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103085 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103087 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103090 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103092 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103095 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103098 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103100 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103103 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103105 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103108 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103110 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103113 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103115 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103118 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103122 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103124 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103127 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103129 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.112373 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103132 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103134 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103137 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103139 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103143 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103146 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103148 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103151 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103153 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103156 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103159 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103161 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103163 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103166 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103169 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103171 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103174 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103176 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103179 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103181 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.112853 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103184 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103186 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103189 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103192 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103194 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103197 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103199 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103202 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103206 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103208 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103210 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103213 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103215 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103218 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103221 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103223 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.113337 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.103250 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103287 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103292 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103295 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103298 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103301 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103304 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103306 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103309 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103311 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103315 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103317 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103320 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103322 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103325 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103327 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103330 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103332 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103335 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.113715 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103337 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103339 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103342 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103344 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103347 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103350 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103353 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103355 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103357 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103360 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103362 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103365 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103367 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103370 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103373 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103375 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103378 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103380 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103383 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103386 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.114165 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103389 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103391 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103394 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103396 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103399 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103402 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103404 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103407 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103409 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103411 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103414 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103416 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103419 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103422 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103424 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103429 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103433 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103436 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103438 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.114635 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103441 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103443 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103446 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103448 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103451 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103454 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103457 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103460 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103462 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103465 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103467 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103470 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103472 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103475 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103477 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103479 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103482 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103485 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103487 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103490 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.115110 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103492 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103494 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103497 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103499 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103502 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103505 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103509 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103512 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103515 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103520 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103523 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103525 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103528 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103530 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103533 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103535 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103538 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103540 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103543 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.115568 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103546 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103548 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103551 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103554 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103556 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103559 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103562 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103564 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.103567 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.111693 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.111710 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.112492 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.112498 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112558 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112563 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112567 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112570 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112572 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112575 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.116079 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112577 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112580 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112582 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112585 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112588 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112590 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112593 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112596 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112598 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112601 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112604 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112606 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112609 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112612 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112615 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112617 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112620 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112623 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112626 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112629 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.116550 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112631 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112634 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112637 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112640 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112643 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112645 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112648 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112650 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112653 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112657 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112661 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112664 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112666 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112669 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112672 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112674 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112677 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112679 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112682 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.117036 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112685 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112687 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112690 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112693 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112695 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112698 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112701 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112704 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112706 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112709 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112711 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112714 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112716 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112718 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112721 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112724 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112727 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112730 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112733 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112735 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.117570 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112738 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112741 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112743 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112746 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112748 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112752 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112756 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112759 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112762 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112764 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112767 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112770 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112772 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112774 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112778 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112791 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112794 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112797 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112800 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.118054 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112802 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112805 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112807 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112810 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112813 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112815 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112818 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112820 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112824 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112826 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112829 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112831 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112834 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112837 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112839 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112842 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112844 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112847 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112849 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.118590 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112852 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112854 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.112857 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.112884 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113041 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113047 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113051 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113056 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113059 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113062 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113065 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113068 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113070 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113073 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113075 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.119050 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113078 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113080 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113082 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113085 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113087 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113090 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113093 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113096 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113099 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113101 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113104 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113106 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113109 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113111 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113114 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113117 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113119 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113122 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113124 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.119414 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113126 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113129 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113132 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113134 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113136 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113139 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113141 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113143 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113147 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113149 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113152 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113154 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113157 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113159 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113162 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113164 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113166 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113169 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113171 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113174 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.119943 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113177 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113179 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113182 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113185 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113187 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113192 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113195 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113197 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113200 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113202 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113204 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113207 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113209 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113212 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113214 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113217 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113219 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113221 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113224 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.120435 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113226 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113230 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113235 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113238 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113240 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113243 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113245 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113248 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113250 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113252 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113255 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113258 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113261 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113263 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113266 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113268 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113271 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113273 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.120918 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113276 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113278 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113280 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113283 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113285 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113288 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113291 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113293 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113296 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113298 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113300 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113303 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113305 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113308 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113310 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113313 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.121667 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113316 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.113339 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113370 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113374 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113376 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113380 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113382 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113385 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113387 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113390 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113393 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113395 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113398 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113400 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113403 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113405 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:57:54.122247 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113408 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113410 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113413 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113415 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113418 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113420 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113422 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113425 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113427 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113430 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113434 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113437 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113441 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113445 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113449 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113451 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113457 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113459 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:57:54.122671 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113461 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113464 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113466 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113469 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113471 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113474 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113476 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113479 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113481 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113484 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113486 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113488 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113491 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113493 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113496 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113498 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113501 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113503 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113505 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:57:54.123304 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113508 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113510 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113512 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113515 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113517 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113519 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113522 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113524 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113526 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113529 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113531 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113535 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113540 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113542 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113545 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113547 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113549 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113553 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113555 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:57:54.123985 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113558 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113560 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113562 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113565 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113567 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113570 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113572 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113575 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113578 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113580 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113582 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113585 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113587 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113590 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113592 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113595 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113597 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113599 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113602 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113604 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:57:54.124449 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113606 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113609 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113611 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113613 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113618 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113620 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113622 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113625 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113627 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113630 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113632 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113635 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113637 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:54.113640 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.113694 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.116014 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.117131 2578 server.go:1014] "Starting client certificate rotation" Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.117241 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:57:54.125004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.118088 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:57:54.125418 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.119416 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:57:54.125418 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.124453 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:57:54.136469 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.136452 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:57:54.139614 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.139595 2578 log.go:25] "Validated CRI v1 image API" Oct 30 23:57:54.140510 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.140483 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:57:54.143102 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.143083 2578 fs.go:135] Filesystem UUIDs: map[75511820-59e9-4be9-bade-537393f2b359:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b5516c5b-58df-473c-954c-97caa7a37b96:/dev/nvme0n1p4] Oct 30 23:57:54.143170 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.143102 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:57:54.151213 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.150869 2578 manager.go:217] Machine: {Timestamp:2026-10-30 23:57:54.148677373 +0000 UTC m=+0.361539559 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102022 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23a0ed0ab7f9719b607349b13b2348 SystemUUID:ec23a0ed-0ab7-f971-9b60-7349b13b2348 BootID:c807b218-bcf5-41cb-b775-541c261804bf Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:e2:a4:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:e2:a4:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:39:3a:6c:87:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:57:54.151213 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.151209 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:57:54.151337 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.151321 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:57:54.153832 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.153815 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:57:54.154957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.154935 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:57:54.155083 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.154961 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-246.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:57:54.155123 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.155086 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:57:54.155123 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.155093 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:57:54.155123 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.155118 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:57:54.156063 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.156053 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:57:54.157271 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.157261 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:57:54.161635 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161624 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:57:54.161746 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161737 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:57:54.161805 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161768 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:57:54.161805 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161796 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:57:54.161886 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161814 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:57:54.161886 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161826 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:57:54.161886 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.161834 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:57:54.162183 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.162158 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:57:54.164669 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.164651 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:57:54.171319 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.171305 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:57:54.172225 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172214 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:57:54.172275 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172244 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:57:54.172391 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172381 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:57:54.172423 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172400 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:57:54.172423 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172406 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:57:54.172423 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172411 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:57:54.172423 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172417 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:57:54.172423 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.172423 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:57:54.173243 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.173235 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:57:54.173243 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.173244 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:57:54.173301 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.173252 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:57:54.173301 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.173258 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:57:54.173301 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.173266 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:57:54.173301 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.173275 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:57:54.174303 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.174294 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:57:54.174332 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.174303 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:57:54.178392 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.178379 2578 server.go:1263] "Started kubelet" Oct 30 23:57:54.178487 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.178465 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:57:54.178603 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.178531 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:57:54.178634 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.178625 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:57:54.179022 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.178990 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-246.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:57:54.179205 ip-10-0-140-246 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:57:54.180059 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.180043 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:57:54.181710 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.181696 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:57:54.184213 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.184189 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:57:54.184213 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.184196 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-246.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:57:54.184805 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.183699 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-246.us-east-2.compute.internal.18e3740c9eec7544 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-246.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-246.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:57:54.178315588 +0000 UTC m=+0.391177774,LastTimestamp:2026-10-30 23:57:54.178315588 +0000 UTC m=+0.391177774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-246.us-east-2.compute.internal,}" Oct 30 23:57:54.185256 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.185240 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:57:54.187045 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.187023 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:57:54.187688 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.187670 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:57:54.187820 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.187772 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.188450 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188433 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:57:54.188450 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188443 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:57:54.188450 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188451 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:57:54.188596 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188552 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:57:54.188596 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188562 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:57:54.188657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188633 2578 factory.go:55] Registering systemd factory Oct 30 23:57:54.188657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.188648 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:57:54.189079 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.189066 2578 factory.go:153] Registering CRI-O factory Oct 30 23:57:54.189124 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.189083 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 23:57:54.189161 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.189146 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:57:54.189220 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.189209 2578 factory.go:103] Registering Raw factory Oct 30 23:57:54.189251 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.189228 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 23:57:54.190481 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.190380 2578 manager.go:319] Starting recovery of all containers Oct 30 23:57:54.190942 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.190918 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-246.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:57:54.196730 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.196701 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 23:57:54.197441 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.197400 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:57:54.202990 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.202969 2578 manager.go:324] Recovery completed Oct 30 23:57:54.203515 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.203493 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6sswp" Oct 30 23:57:54.207464 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.207450 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.209161 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209143 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.209218 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209174 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.209218 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209189 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.209655 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209642 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:57:54.209655 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209652 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:57:54.209745 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209666 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:57:54.209745 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.209643 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.210383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.210370 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.210437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.210392 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.210437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.210403 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.211151 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.211090 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-246.us-east-2.compute.internal.18e3740ca0c3244f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-246.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-246.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-246.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:57:54.209162319 +0000 UTC m=+0.422024515,LastTimestamp:2026-10-30 23:57:54.209162319 +0000 UTC m=+0.422024515,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-246.us-east-2.compute.internal,}" Oct 30 23:57:54.211830 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.211819 2578 policy_none.go:50] "Start" Oct 30 23:57:54.211866 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.211834 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:57:54.211866 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.211847 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:57:54.213809 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.213799 2578 policy_none.go:44] "Start" Oct 30 23:57:54.215529 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.215514 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6sswp" Oct 30 23:57:54.246398 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.246376 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:57:54.248692 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.248678 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 23:57:54.248769 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.248703 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:57:54.248769 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.248711 2578 server.go:87] "Starting device plugin registration server" Oct 30 23:57:54.248959 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.248946 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:57:54.249007 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.248961 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:57:54.249059 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.249022 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:57:54.249108 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.249097 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:57:54.249108 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.249106 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:57:54.249567 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.249552 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:57:54.249629 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.249588 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.279066 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.275861 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:57:54.279066 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.275967 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:57:54.279066 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.275982 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:57:54.279066 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.276010 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:57:54.281569 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.281555 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:54.349872 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.349828 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.351661 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.351646 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.351735 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.351673 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.351735 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.351684 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.351735 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.351705 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.362033 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.362017 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.362078 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.362037 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-246.us-east-2.compute.internal\": node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.376253 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.376233 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal"] Oct 30 23:57:54.376326 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.376281 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.377016 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.376993 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.377016 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.377016 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.377126 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.377025 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.378964 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.378952 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.379129 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.379192 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379154 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.379651 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379635 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.379714 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379636 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.379714 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379686 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.379714 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379696 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.379833 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379667 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.379833 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.379743 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.381747 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.381733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.381819 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.381756 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:57:54.382441 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.382427 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:57:54.382528 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.382454 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:57:54.382528 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.382468 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:57:54.389897 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.389878 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.409061 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.409040 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.411592 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.411577 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.489582 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.489556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.489657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.489585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.489657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.489604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc47763c26c267f0abea2ce914234e24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"fc47763c26c267f0abea2ce914234e24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.490619 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.490603 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.589932 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.589914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.589988 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.589938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.589988 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.589956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc47763c26c267f0abea2ce914234e24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"fc47763c26c267f0abea2ce914234e24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.590050 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.590002 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc47763c26c267f0abea2ce914234e24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"fc47763c26c267f0abea2ce914234e24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.590050 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.590010 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.590050 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.590023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.590994 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.590967 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.691680 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.691636 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.713808 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.713778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.716928 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:54.716919 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:54.792627 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.792603 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.893026 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.893004 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:54.993542 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:54.993497 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:55.093902 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:55.093882 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:55.117046 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.117028 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:57:55.117159 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.117143 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:57:55.194641 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:55.194617 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:55.221365 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.221322 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:52:54 +0000 UTC" deadline="2028-05-20 03:49:50.318735663 +0000 UTC" Oct 30 23:57:55.221365 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.221364 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13611h51m55.097374575s" Oct 30 23:57:55.295068 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:55.295012 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:57:55.313571 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.313551 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:55.336865 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:55.336827 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc47763c26c267f0abea2ce914234e24.slice/crio-1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a WatchSource:0}: Error finding container 1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a: Status 404 returned error can't find the container with id 1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a Oct 30 23:57:55.337229 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:55.337211 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aab21424359483efbf6437e8de7b4f.slice/crio-39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d WatchSource:0}: Error finding container 39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d: Status 404 returned error can't find the container with id 39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d Oct 30 23:57:55.343087 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.343072 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:57:55.388828 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.388809 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:55.398208 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.398181 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:57:55.400214 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.400202 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:57:55.416053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.416026 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:57:55.736662 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.736604 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:55.758474 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:55.758456 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:56.162692 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.162670 2578 apiserver.go:52] "Watching apiserver" Oct 30 23:57:56.170439 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.170412 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:57:56.170848 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.170828 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4qbw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc","openshift-multus/network-metrics-daemon-dzqgd","openshift-ovn-kubernetes/ovnkube-node-w75k7","openshift-dns/node-resolver-cjnpt","openshift-image-registry/node-ca-n28vw","openshift-multus/multus-wk7mn","openshift-multus/multus-additional-cni-plugins-2fpcf","openshift-network-diagnostics/network-check-target-6gwvc","openshift-network-operator/iptables-alerter-jm9dj","kube-system/global-pull-secret-syncer-qkj2v","kube-system/konnectivity-agent-v79mw"] Oct 30 23:57:56.173241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.173224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.175002 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.174985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.175086 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.175065 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:57:56.177244 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.176938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.177244 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.176955 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.177244 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.177077 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.177244 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.177138 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpmk\"" Oct 30 23:57:56.178851 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.178835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.180876 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.180856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.180973 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.180924 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:57:56.183682 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.183661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.187493 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.187374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.188388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.187948 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:57:56.188981 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.188958 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.191458 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.191440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.193359 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.193341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.195489 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.195470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:56.195573 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.195543 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:57:56.197512 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.197497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.198376 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wng4p\" (UniqueName: \"kubernetes.io/projected/3db5e05a-532e-4937-a241-3a995c5c43ae-kube-api-access-wng4p\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.198376 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-kubelet-config\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.198513 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cni-binary-copy\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.198513 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71c479d0-cb02-4549-acb7-3a206eb2ecb7-serviceca\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.198513 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7sgf\" (UniqueName: \"kubernetes.io/projected/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kube-api-access-l7sgf\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.198653 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cnibin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.198653 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-hostroot\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.198653 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-multus-certs\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.198653 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdchj\" (UniqueName: \"kubernetes.io/projected/b613d66f-e947-427e-9694-3f0b609f35c4-kube-api-access-jdchj\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-ovn\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-script-lib\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-registration-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-slash\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-daemon-config\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-systemd-units\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.198948 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-config\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-system-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.198990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb57k\" (UniqueName: \"kubernetes.io/projected/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-kube-api-access-pb57k\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b613d66f-e947-427e-9694-3f0b609f35c4-tmp-dir\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-systemd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86e12f20-75a3-480a-81b1-f6bf080a6733-agent-certs\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-bin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgdj\" (UniqueName: \"kubernetes.io/projected/71c479d0-cb02-4549-acb7-3a206eb2ecb7-kube-api-access-9pgdj\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-node-log\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-netns\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.199284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-device-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-socket-dir-parent\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-kubelet\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-log-socket\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db5e05a-532e-4937-a241-3a995c5c43ae-ovn-node-metrics-cert\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199410 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-sys-fs\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-env-overrides\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86e12f20-75a3-480a-81b1-f6bf080a6733-konnectivity-ca\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-var-lib-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-bin\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-etc-kubernetes\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.199975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-etc-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-netd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr8p8\" (UniqueName: \"kubernetes.io/projected/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-kube-api-access-lr8p8\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-socket-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-kubelet\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-netns\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-k8s-cni-cncf-io\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.199990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c479d0-cb02-4549-acb7-3a206eb2ecb7-host\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.200013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b613d66f-e947-427e-9694-3f0b609f35c4-hosts-file\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.200072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-dbus\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.200094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-multus\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.200657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.200117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-conf-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.203894 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.203877 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:57:56.222881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.222862 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8bhpv" Oct 30 23:57:56.233748 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.233730 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8bhpv" Oct 30 23:57:56.269061 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.269000 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:57:56.269138 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.269085 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dtg7z\"" Oct 30 23:57:56.269138 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.269121 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g645g\"" Oct 30 23:57:56.269208 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.269166 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ccbg\"" Oct 30 23:57:56.269208 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.269180 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:57:56.269266 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.269219 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:57:56.270083 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270063 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6ff8s\"" Oct 30 23:57:56.270201 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270125 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqcpl\"" Oct 30 23:57:56.270201 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270150 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r8kbq\"" Oct 30 23:57:56.270201 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270178 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrv6t\"" Oct 30 23:57:56.270357 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270225 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72z8g\"" Oct 30 23:57:56.270357 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270277 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:57:56.270357 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270275 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:57:56.270357 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270298 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.270357 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270294 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:57:56.270357 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270351 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.270533 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270373 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.270533 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270385 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:57:56.270918 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270900 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.270994 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.270977 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.271079 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271066 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:57:56.271144 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271120 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.271202 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271188 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.271270 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271255 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:57:56.271378 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271310 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.271535 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.271599 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271562 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:57:56.271648 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271624 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:57:56.271687 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271656 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:57:56.271728 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271687 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:57:56.271758 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.271750 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.272138 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.272121 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:57:56.272138 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.272128 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:57:56.282215 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.282174 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"29aab21424359483efbf6437e8de7b4f","Type":"ContainerStarted","Data":"39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d"} Oct 30 23:57:56.283268 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.283237 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"fc47763c26c267f0abea2ce914234e24","Type":"ContainerStarted","Data":"1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a"} Oct 30 23:57:56.290053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.290036 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:57:56.301298 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c199e89-40d7-4a83-815c-f3533fcc31c4-host-slash\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.301298 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng4p\" (UniqueName: \"kubernetes.io/projected/3db5e05a-532e-4937-a241-3a995c5c43ae-kube-api-access-wng4p\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.301298 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-kubelet-config\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.301298 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-system-cni-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.301504 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:56.301504 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-kubelet-config\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.301504 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cni-binary-copy\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.301504 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71c479d0-cb02-4549-acb7-3a206eb2ecb7-serviceca\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.301504 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.301665 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-lib-modules\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.301665 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sgf\" (UniqueName: \"kubernetes.io/projected/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kube-api-access-l7sgf\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.301665 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cnibin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.301665 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-hostroot\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.301665 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301633 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cnibin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.301864 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301850 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-hostroot\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.301916 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-multus-certs\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.301916 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdchj\" (UniqueName: \"kubernetes.io/projected/b613d66f-e947-427e-9694-3f0b609f35c4-kube-api-access-jdchj\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.302008 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-ovn\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.302008 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.301992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-script-lib\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.302091 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.302091 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-registration-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.302174 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-slash\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.302174 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cni-binary-copy\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.302174 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-var-lib-kubelet\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.302174 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-registration-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.302174 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-daemon-config\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-systemd-units\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-kubernetes\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-sys\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302309 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67z2d\" (UniqueName: \"kubernetes.io/projected/e3036952-ce27-471a-9fa5-933ea133e22e-kube-api-access-67z2d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-config\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-system-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.302383 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb57k\" (UniqueName: \"kubernetes.io/projected/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-kube-api-access-pb57k\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-daemon-config\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b613d66f-e947-427e-9694-3f0b609f35c4-tmp-dir\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-systemd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86e12f20-75a3-480a-81b1-f6bf080a6733-agent-certs\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-tmp\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-bin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgdj\" (UniqueName: \"kubernetes.io/projected/71c479d0-cb02-4549-acb7-3a206eb2ecb7-kube-api-access-9pgdj\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.302734 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-system-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-systemd-units\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-ovn\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-node-log\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysconfig\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-multus-certs\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-netns\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303078 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.302990 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-node-log\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-netns\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-bin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.303180 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-systemd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-slash\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.303238 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:56.803219387 +0000 UTC m=+3.016081580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-systemd\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-etc-tuned\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-device-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-socket-dir-parent\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303479 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-device-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-socket-dir-parent\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-kubelet\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71c479d0-cb02-4549-acb7-3a206eb2ecb7-serviceca\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-log-socket\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-kubelet\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db5e05a-532e-4937-a241-3a995c5c43ae-ovn-node-metrics-cert\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-log-socket\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-sys-fs\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-env-overrides\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-cnibin\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6sbf\" (UniqueName: \"kubernetes.io/projected/fbea15b0-87b2-480b-9bb7-496f71990632-kube-api-access-j6sbf\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-modprobe-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.303957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86e12f20-75a3-480a-81b1-f6bf080a6733-konnectivity-ca\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-run\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.303991 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-var-lib-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.304027 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-bin\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-etc-kubernetes\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-etc-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-host\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-netd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8p8\" (UniqueName: \"kubernetes.io/projected/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-kube-api-access-lr8p8\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-socket-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-kubelet\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-netns\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c199e89-40d7-4a83-815c-f3533fcc31c4-iptables-alerter-script\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4m8w\" (UniqueName: \"kubernetes.io/projected/4c199e89-40d7-4a83-815c-f3533fcc31c4-kube-api-access-x4m8w\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.304681 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-k8s-cni-cncf-io\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c479d0-cb02-4549-acb7-3a206eb2ecb7-host\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b613d66f-e947-427e-9694-3f0b609f35c4-tmp-dir\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b613d66f-e947-427e-9694-3f0b609f35c4-hosts-file\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b613d66f-e947-427e-9694-3f0b609f35c4-hosts-file\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.304589 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret podName:106281ea-1622-4966-bfd3-bdba5d770f5f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:56.804573536 +0000 UTC m=+3.017435708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret") pod "global-pull-secret-syncer-qkj2v" (UID: "106281ea-1622-4966-bfd3-bdba5d770f5f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304606 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-netd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-dbus\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-config\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-conf\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-multus\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-conf-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-bin\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304747 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-k8s-cni-cncf-io\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305388 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c479d0-cb02-4549-acb7-3a206eb2ecb7-host\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304964 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-etc-kubernetes\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304964 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-multus\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-var-lib-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.304999 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-netns\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-sys-fs\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-socket-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-dbus\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-conf-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305126 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86e12f20-75a3-480a-81b1-f6bf080a6733-konnectivity-ca\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-etc-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305159 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-kubelet\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.305997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.305761 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-script-lib\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.306745 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.306707 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-env-overrides\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.308103 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.308086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db5e05a-532e-4937-a241-3a995c5c43ae-ovn-node-metrics-cert\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.308405 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.308383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86e12f20-75a3-480a-81b1-f6bf080a6733-agent-certs\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.313162 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.313137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng4p\" (UniqueName: \"kubernetes.io/projected/3db5e05a-532e-4937-a241-3a995c5c43ae-kube-api-access-wng4p\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.314122 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.313724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sgf\" (UniqueName: \"kubernetes.io/projected/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kube-api-access-l7sgf\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.314122 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.314070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdchj\" (UniqueName: \"kubernetes.io/projected/b613d66f-e947-427e-9694-3f0b609f35c4-kube-api-access-jdchj\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.314723 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.314679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb57k\" (UniqueName: \"kubernetes.io/projected/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-kube-api-access-pb57k\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.315341 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.315323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgdj\" (UniqueName: \"kubernetes.io/projected/71c479d0-cb02-4549-acb7-3a206eb2ecb7-kube-api-access-9pgdj\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.316860 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.316836 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8p8\" (UniqueName: \"kubernetes.io/projected/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-kube-api-access-lr8p8\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.405954 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.405928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-tmp\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406099 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.405968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysconfig\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406099 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-systemd\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406099 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-etc-tuned\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406099 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-cnibin\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.406249 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.406249 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6sbf\" (UniqueName: \"kubernetes.io/projected/fbea15b0-87b2-480b-9bb7-496f71990632-kube-api-access-j6sbf\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.406249 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-modprobe-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406249 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-run\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406249 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-host\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c199e89-40d7-4a83-815c-f3533fcc31c4-iptables-alerter-script\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-cnibin\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4m8w\" (UniqueName: \"kubernetes.io/projected/4c199e89-40d7-4a83-815c-f3533fcc31c4-kube-api-access-x4m8w\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-conf\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406339 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-run\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-host\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c199e89-40d7-4a83-815c-f3533fcc31c4-host-slash\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c199e89-40d7-4a83-815c-f3533fcc31c4-host-slash\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-system-cni-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:56.406455 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406449 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysconfig\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-lib-modules\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-var-lib-kubelet\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-kubernetes\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-sys\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z2d\" (UniqueName: \"kubernetes.io/projected/e3036952-ce27-471a-9fa5-933ea133e22e-kube-api-access-67z2d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-modprobe-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406857 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-system-cni-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.407010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.406491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-systemd\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-var-lib-kubelet\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-conf\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407063 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-kubernetes\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407437 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407101 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-sys\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407654 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c199e89-40d7-4a83-815c-f3533fcc31c4-iptables-alerter-script\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.407654 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.407654 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-lib-modules\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.407832 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.407832 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.407737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.408265 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.408153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.408677 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.408652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-tmp\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.410163 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.410134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-etc-tuned\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.420314 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.420296 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:57:56.420314 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.420316 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:57:56.420314 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.420326 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xvx55 for pod openshift-network-diagnostics/network-check-target-6gwvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:57:56.420497 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.420373 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55 podName:b31374f9-c1aa-4288-a5db-c88ec5b3f732 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:56.920361446 +0000 UTC m=+3.133223629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xvx55" (UniqueName: "kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55") pod "network-check-target-6gwvc" (UID: "b31374f9-c1aa-4288-a5db-c88ec5b3f732") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:57:56.422904 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.422881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6sbf\" (UniqueName: \"kubernetes.io/projected/fbea15b0-87b2-480b-9bb7-496f71990632-kube-api-access-j6sbf\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.422985 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.422948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z2d\" (UniqueName: \"kubernetes.io/projected/e3036952-ce27-471a-9fa5-933ea133e22e-kube-api-access-67z2d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.422985 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.422977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4m8w\" (UniqueName: \"kubernetes.io/projected/4c199e89-40d7-4a83-815c-f3533fcc31c4-kube-api-access-x4m8w\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.486916 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.486885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:57:56.496579 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.496556 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb613d66f_e947_427e_9694_3f0b609f35c4.slice/crio-381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2 WatchSource:0}: Error finding container 381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2: Status 404 returned error can't find the container with id 381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2 Oct 30 23:57:56.496693 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.496646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:57:56.505716 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.505694 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e12f20_75a3_480a_81b1_f6bf080a6733.slice/crio-ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356 WatchSource:0}: Error finding container ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356: Status 404 returned error can't find the container with id ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356 Oct 30 23:57:56.510703 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.510559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:57:56.520844 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.520216 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e72e0f4_7067_409f_bc3a_0dcff8e656fa.slice/crio-8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912 WatchSource:0}: Error finding container 8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912: Status 404 returned error can't find the container with id 8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912 Oct 30 23:57:56.527688 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.527669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wk7mn" Oct 30 23:57:56.531301 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.531284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:57:56.534538 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.534520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:57:56.537208 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.537067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:57:56.542913 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.542896 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db5e05a_532e_4937_a241_3a995c5c43ae.slice/crio-749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90 WatchSource:0}: Error finding container 749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90: Status 404 returned error can't find the container with id 749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90 Oct 30 23:57:56.546633 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.546577 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a607b81_bbfa_4a4e_9828_2065d1e3bf49.slice/crio-d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709 WatchSource:0}: Error finding container d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709: Status 404 returned error can't find the container with id d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709 Oct 30 23:57:56.549413 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.549386 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c479d0_cb02_4549_acb7_3a206eb2ecb7.slice/crio-19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f WatchSource:0}: Error finding container 19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f: Status 404 returned error can't find the container with id 19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f Oct 30 23:57:56.557671 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.557653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:57:56.559762 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.559746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:57:56.567587 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.567565 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c199e89_40d7_4a83_815c_f3533fcc31c4.slice/crio-159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8 WatchSource:0}: Error finding container 159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8: Status 404 returned error can't find the container with id 159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8 Oct 30 23:57:56.569529 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:57:56.569506 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbea15b0_87b2_480b_9bb7_496f71990632.slice/crio-21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e WatchSource:0}: Error finding container 21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e: Status 404 returned error can't find the container with id 21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e Oct 30 23:57:56.716319 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.716292 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:57:56.808499 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.808472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:56.808566 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:56.808513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:56.808623 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.808606 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:56.808669 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.808661 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:57.808647536 +0000 UTC m=+4.021509707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:56.808711 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.808608 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:56.808764 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:56.808753 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret podName:106281ea-1622-4966-bfd3-bdba5d770f5f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:57.808739171 +0000 UTC m=+4.021601354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret") pod "global-pull-secret-syncer-qkj2v" (UID: "106281ea-1622-4966-bfd3-bdba5d770f5f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:57.009890 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.009813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:57.010040 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.010024 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:57:57.010090 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.010045 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:57:57.010090 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.010057 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xvx55 for pod openshift-network-diagnostics/network-check-target-6gwvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:57:57.010143 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.010112 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55 podName:b31374f9-c1aa-4288-a5db-c88ec5b3f732 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:58.010094943 +0000 UTC m=+4.222957129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvx55" (UniqueName: "kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55") pod "network-check-target-6gwvc" (UID: "b31374f9-c1aa-4288-a5db-c88ec5b3f732") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:57:57.237247 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.236931 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:52:56 +0000 UTC" deadline="2028-08-16 10:46:22.51544092 +0000 UTC" Oct 30 23:57:57.237247 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.237187 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15730h48m25.278260323s" Oct 30 23:57:57.277221 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.276497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:57.277221 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.276635 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:57:57.277221 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.277072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:57.277221 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.277172 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:57:57.293334 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.293303 2578 generic.go:356] "Generic (PLEG): container finished" podID="29aab21424359483efbf6437e8de7b4f" containerID="64fa0e1f4c8ca38b40ce07d3fd07b2c198c93432eb49dc9750b69cec53fb95a5" exitCode=0 Oct 30 23:57:57.293449 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.293391 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"29aab21424359483efbf6437e8de7b4f","Type":"ContainerDied","Data":"64fa0e1f4c8ca38b40ce07d3fd07b2c198c93432eb49dc9750b69cec53fb95a5"} Oct 30 23:57:57.306556 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.306509 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"fc47763c26c267f0abea2ce914234e24","Type":"ContainerStarted","Data":"9f12bae981c0cc607b3dc403511c5657c8967fd380328205e04ad2c5b998e1f6"} Oct 30 23:57:57.320511 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.320488 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerStarted","Data":"21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e"} Oct 30 23:57:57.324737 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.324714 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jm9dj" event={"ID":"4c199e89-40d7-4a83-815c-f3533fcc31c4","Type":"ContainerStarted","Data":"159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8"} Oct 30 23:57:57.329773 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.329719 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n28vw" event={"ID":"71c479d0-cb02-4549-acb7-3a206eb2ecb7","Type":"ContainerStarted","Data":"19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f"} Oct 30 23:57:57.334828 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.334771 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wk7mn" event={"ID":"8a607b81-bbfa-4a4e-9828-2065d1e3bf49","Type":"ContainerStarted","Data":"d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709"} Oct 30 23:57:57.338493 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.338471 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjnpt" event={"ID":"b613d66f-e947-427e-9694-3f0b609f35c4","Type":"ContainerStarted","Data":"381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2"} Oct 30 23:57:57.355359 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.355323 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" event={"ID":"e3036952-ce27-471a-9fa5-933ea133e22e","Type":"ContainerStarted","Data":"03c99e89b793ddef76883f11a602e8a672f4e6329dfb6e37977547ab50c82b22"} Oct 30 23:57:57.362150 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.362125 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90"} Oct 30 23:57:57.374991 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.374964 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" event={"ID":"1e72e0f4-7067-409f-bc3a-0dcff8e656fa","Type":"ContainerStarted","Data":"8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912"} Oct 30 23:57:57.397902 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.397873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v79mw" event={"ID":"86e12f20-75a3-480a-81b1-f6bf080a6733","Type":"ContainerStarted","Data":"ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356"} Oct 30 23:57:57.820970 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.820198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:57.820970 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:57.820248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:57.820970 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.820374 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:57.820970 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.820431 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret podName:106281ea-1622-4966-bfd3-bdba5d770f5f nodeName:}" failed. No retries permitted until 2026-10-30 23:57:59.820413619 +0000 UTC m=+6.033275807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret") pod "global-pull-secret-syncer-qkj2v" (UID: "106281ea-1622-4966-bfd3-bdba5d770f5f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:57.820970 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.820843 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:57.820970 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:57.820895 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:57:59.820879819 +0000 UTC m=+6.033742004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:58.022260 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:58.022037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:58.022260 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:58.022230 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:57:58.022260 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:58.022251 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:57:58.022260 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:58.022262 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xvx55 for pod openshift-network-diagnostics/network-check-target-6gwvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:57:58.022559 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:58.022313 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55 podName:b31374f9-c1aa-4288-a5db-c88ec5b3f732 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:00.022298224 +0000 UTC m=+6.235160409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvx55" (UniqueName: "kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55") pod "network-check-target-6gwvc" (UID: "b31374f9-c1aa-4288-a5db-c88ec5b3f732") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:57:58.237966 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:58.237882 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:52:56 +0000 UTC" deadline="2028-07-20 10:44:29.247808854 +0000 UTC" Oct 30 23:57:58.237966 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:58.237920 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15082h46m31.009892743s" Oct 30 23:57:58.281179 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:58.281150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:58.281328 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:58.281287 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:57:58.417805 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:58.417706 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"29aab21424359483efbf6437e8de7b4f","Type":"ContainerStarted","Data":"703ec8608bc6d0e26513bda7e71f6cf7c16ec9cb45f1acff5611477115891841"} Oct 30 23:57:58.433607 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:58.433553 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" podStartSLOduration=3.433535715 podStartE2EDuration="3.433535715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:57.3292281 +0000 UTC m=+3.542090297" watchObservedRunningTime="2026-10-30 23:57:58.433535715 +0000 UTC m=+4.646397909" Oct 30 23:57:59.276633 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:59.276602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:59.277074 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:59.276738 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:57:59.277145 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:59.277099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:57:59.277205 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:59.277181 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:57:59.427441 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:59.427295 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerStarted","Data":"716947db0ad59cd9daa3581e2e28533f8f0820d096f9bb45f7328522b863001f"} Oct 30 23:57:59.454249 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:59.453174 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" podStartSLOduration=4.4531568 podStartE2EDuration="4.4531568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:58.434554467 +0000 UTC m=+4.647416666" watchObservedRunningTime="2026-10-30 23:57:59.4531568 +0000 UTC m=+5.666018972" Oct 30 23:57:59.840449 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:59.839536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:57:59.840449 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:57:59.839652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:57:59.840449 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:59.839810 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:59.840449 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:59.839878 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:03.839857537 +0000 UTC m=+10.052719724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:57:59.840449 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:59.840285 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:57:59.840449 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:57:59.840337 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret podName:106281ea-1622-4966-bfd3-bdba5d770f5f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:03.840321744 +0000 UTC m=+10.053183933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret") pod "global-pull-secret-syncer-qkj2v" (UID: "106281ea-1622-4966-bfd3-bdba5d770f5f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:00.042573 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:00.041927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:00.042573 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:00.042096 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:00.042573 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:00.042118 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:00.042573 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:00.042131 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xvx55 for pod openshift-network-diagnostics/network-check-target-6gwvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:00.042573 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:00.042185 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55 podName:b31374f9-c1aa-4288-a5db-c88ec5b3f732 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:04.042168155 +0000 UTC m=+10.255030335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvx55" (UniqueName: "kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55") pod "network-check-target-6gwvc" (UID: "b31374f9-c1aa-4288-a5db-c88ec5b3f732") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:00.279680 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:00.279170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:00.279680 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:00.279314 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:00.431271 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:00.430523 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v79mw" event={"ID":"86e12f20-75a3-480a-81b1-f6bf080a6733","Type":"ContainerStarted","Data":"d2d200a072545b43f41a41a4bc9abd1095533826ffcde586a93b06304eaa1f95"} Oct 30 23:58:00.434392 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:00.434357 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="716947db0ad59cd9daa3581e2e28533f8f0820d096f9bb45f7328522b863001f" exitCode=0 Oct 30 23:58:00.434532 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:00.434410 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerDied","Data":"716947db0ad59cd9daa3581e2e28533f8f0820d096f9bb45f7328522b863001f"} Oct 30 23:58:00.449421 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:00.448560 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v79mw" podStartSLOduration=2.845337157 podStartE2EDuration="6.448544813s" totalImagesPullingTime="3.603207656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:00.448251261 +0000 UTC m=+6.661113456" watchObservedRunningTime="2026-10-30 23:58:00.448544813 +0000 UTC m=+6.661407008" Oct 30 23:58:01.276903 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:01.276258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:01.276903 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:01.276395 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:01.276903 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:01.276464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:01.276903 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:01.276533 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:02.287039 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:02.286970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:02.287505 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:02.287177 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:02.442820 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:02.442730 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" event={"ID":"1e72e0f4-7067-409f-bc3a-0dcff8e656fa","Type":"ContainerStarted","Data":"68c381897baa2a9f49a73dd47b2947ad1c0e5f0df4fdb7bea1dc441fcb75cb91"} Oct 30 23:58:02.444897 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:02.444867 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n28vw" event={"ID":"71c479d0-cb02-4549-acb7-3a206eb2ecb7","Type":"ContainerStarted","Data":"93f89ca6f4748968c09ef69b4fd4840752e4a157d2f6dba555fb26a141eda27f"} Oct 30 23:58:03.276854 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:03.276161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:03.276854 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:03.276283 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:03.276854 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:03.276649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:03.276854 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:03.276737 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:03.655141 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:03.655115 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:58:03.873989 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:03.873309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:03.873989 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:03.873356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:03.873989 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:03.873499 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:03.873989 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:03.873558 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret podName:106281ea-1622-4966-bfd3-bdba5d770f5f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:11.873540072 +0000 UTC m=+18.086402245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret") pod "global-pull-secret-syncer-qkj2v" (UID: "106281ea-1622-4966-bfd3-bdba5d770f5f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:03.873989 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:03.873625 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:03.873989 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:03.873661 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:11.873649 +0000 UTC m=+18.086511182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:04.074333 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.074292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:04.074487 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:04.074460 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:04.074487 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:04.074483 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:04.074556 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:04.074496 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xvx55 for pod openshift-network-diagnostics/network-check-target-6gwvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:04.074588 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:04.074560 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55 podName:b31374f9-c1aa-4288-a5db-c88ec5b3f732 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:12.074539258 +0000 UTC m=+18.287401449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvx55" (UniqueName: "kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55") pod "network-check-target-6gwvc" (UID: "b31374f9-c1aa-4288-a5db-c88ec5b3f732") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:04.263713 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.262833 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:58:03.655136134Z","UUID":"cbd56675-1060-410a-b16f-9f980d52c714","Handler":null,"Name":"","Endpoint":""} Oct 30 23:58:04.264617 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.264597 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:58:04.264725 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.264625 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:58:04.277050 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.277022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:04.277701 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:04.277673 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:04.453650 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.453611 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wk7mn" event={"ID":"8a607b81-bbfa-4a4e-9828-2065d1e3bf49","Type":"ContainerStarted","Data":"866539028e721352bb4c97b3dd08a2529e89d35f07e05ac38fa2d5401183a662"} Oct 30 23:58:04.460559 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.460532 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" event={"ID":"1e72e0f4-7067-409f-bc3a-0dcff8e656fa","Type":"ContainerStarted","Data":"744974b7602f16f4e8e7787257c0c5248383a086d0272bb229efe23995cd4163"} Oct 30 23:58:04.475303 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.474466 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n28vw" podStartSLOduration=5.55405188 podStartE2EDuration="10.474441996s" totalImagesPullingTime="4.920390116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:02.462637729 +0000 UTC m=+8.675499924" watchObservedRunningTime="2026-10-30 23:58:04.474441996 +0000 UTC m=+10.687304191" Oct 30 23:58:04.475303 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:04.474568 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wk7mn" podStartSLOduration=3.249373743 podStartE2EDuration="10.474561986s" totalImagesPullingTime="7.225188243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:04.474033071 +0000 UTC m=+10.686895266" watchObservedRunningTime="2026-10-30 23:58:04.474561986 +0000 UTC m=+10.687424182" Oct 30 23:58:05.277043 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.276974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:05.277487 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:05.277100 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:05.277487 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.277139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:05.277487 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:05.277209 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:05.420991 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.420730 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:05.421438 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.421406 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:05.464754 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.464702 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjnpt" event={"ID":"b613d66f-e947-427e-9694-3f0b609f35c4","Type":"ContainerStarted","Data":"8837c6cd16af5ba75bdd428f5bc7f99ef679cfce9e5407919982f9095b409141"} Oct 30 23:58:05.467212 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.467179 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" event={"ID":"1e72e0f4-7067-409f-bc3a-0dcff8e656fa","Type":"ContainerStarted","Data":"dbfe7aaa9def715de325e4b349989d747df419234412d4aa74e51b7ddcff6137"} Oct 30 23:58:05.469135 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.469113 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="f253f398afef0f29cd962016af2a4c0b9f86914a1f7fb43457dcb5cd213c498c" exitCode=0 Oct 30 23:58:05.469244 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.469179 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerDied","Data":"f253f398afef0f29cd962016af2a4c0b9f86914a1f7fb43457dcb5cd213c498c"} Oct 30 23:58:05.470291 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.469767 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:05.470291 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.470155 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:05.506198 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.506141 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cjnpt" podStartSLOduration=3.462449827 podStartE2EDuration="11.506123885s" totalImagesPullingTime="8.043674058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:05.482300571 +0000 UTC m=+11.695162768" watchObservedRunningTime="2026-10-30 23:58:05.506123885 +0000 UTC m=+11.718986105" Oct 30 23:58:05.541932 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:05.541845 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" podStartSLOduration=3.495849671 podStartE2EDuration="11.541828045s" totalImagesPullingTime="8.045978374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:05.540907819 +0000 UTC m=+11.753770047" watchObservedRunningTime="2026-10-30 23:58:05.541828045 +0000 UTC m=+11.754690240" Oct 30 23:58:06.277023 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:06.276996 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:06.277168 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:06.277144 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:06.474190 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:06.474151 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="f9c97d076f5eb95851e044e3bb06f40de010c44d014a14843ad644af652c754e" exitCode=0 Oct 30 23:58:06.474376 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:06.474253 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerDied","Data":"f9c97d076f5eb95851e044e3bb06f40de010c44d014a14843ad644af652c754e"} Oct 30 23:58:06.477417 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:06.477384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jm9dj" event={"ID":"4c199e89-40d7-4a83-815c-f3533fcc31c4","Type":"ContainerStarted","Data":"fe94a5de0547db7cf1b2d61e17fd32c6a8c7b3bab4e94e483ba84fa4c305f686"} Oct 30 23:58:06.516491 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:06.516445 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jm9dj" podStartSLOduration=4.450161845 podStartE2EDuration="12.516433944s" totalImagesPullingTime="8.066272099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:06.515915088 +0000 UTC m=+12.728777284" watchObservedRunningTime="2026-10-30 23:58:06.516433944 +0000 UTC m=+12.729296138" Oct 30 23:58:07.276380 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:07.276348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:07.276380 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:07.276381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:07.276609 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:07.276469 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:07.276669 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:07.276603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:07.480969 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:07.480934 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" event={"ID":"e3036952-ce27-471a-9fa5-933ea133e22e","Type":"ContainerStarted","Data":"3bcd0c7db977c15eec06d013ee4b2a2ec3843fd0b53a98912026f2a48968ae59"} Oct 30 23:58:07.500918 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:07.500869 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" podStartSLOduration=3.432295937 podStartE2EDuration="13.500851731s" totalImagesPullingTime="10.068555794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:07.500778191 +0000 UTC m=+13.713640389" watchObservedRunningTime="2026-10-30 23:58:07.500851731 +0000 UTC m=+13.713713928" Oct 30 23:58:08.277196 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:08.277166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:08.277370 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:08.277305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:08.485619 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:08.485585 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="8e1b6f218807fd9da4418d1dd4f997f897d99b433ef27e06c015bb696efd5a79" exitCode=0 Oct 30 23:58:08.486151 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:08.485638 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerDied","Data":"8e1b6f218807fd9da4418d1dd4f997f897d99b433ef27e06c015bb696efd5a79"} Oct 30 23:58:09.277013 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:09.276978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:09.277013 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:09.276995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:09.277228 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:09.277104 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:09.277287 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:09.277225 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:10.277189 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:10.277151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:10.277996 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:10.277291 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:11.276512 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.276362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:11.276609 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.276440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:11.276609 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:11.276593 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:11.276694 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:11.276670 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:11.494741 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.494682 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:11.495108 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495045 2578 generic.go:356] "Generic (PLEG): container finished" podID="3db5e05a-532e-4937-a241-3a995c5c43ae" containerID="8c1c07030f27bd62d0112873e0e92da03390264eccfc3fda7474754c53eaa70f" exitCode=1 Oct 30 23:58:11.495152 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495116 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"62495be2e05a2e7d9d2ac38ba4db9e6eb1b90275df8209ea708d4bc7c2f8ec7c"} Oct 30 23:58:11.495198 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495183 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"4ce6c807778ca6ee6d8386c070af0b09632365c0696e5ce9361fe939506940eb"} Oct 30 23:58:11.495263 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495249 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"a89a1c407c77b9be3bda38e926b69fbe4632169756fa017df21938b9c7c62072"} Oct 30 23:58:11.495309 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495297 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"ae464a16829062bda7e1ba6c5e81e2fbfeaaf4487aaeef59889954e1422e357f"} Oct 30 23:58:11.495355 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495343 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerDied","Data":"8c1c07030f27bd62d0112873e0e92da03390264eccfc3fda7474754c53eaa70f"} Oct 30 23:58:11.495401 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.495388 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"10b872fffdf8976af4112e035932e5f565d87eb1cd09430e257949582294e808"} Oct 30 23:58:11.498076 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.498055 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="7868d83c8ec610ef29397061bce7f09a065050dafb51a6b1de906b9fd19b07d3" exitCode=0 Oct 30 23:58:11.498173 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.498086 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerDied","Data":"7868d83c8ec610ef29397061bce7f09a065050dafb51a6b1de906b9fd19b07d3"} Oct 30 23:58:11.934516 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.934325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:11.934516 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:11.934504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:11.934649 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:11.934452 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:11.934649 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:11.934589 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:11.934649 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:11.934606 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.934592243 +0000 UTC m=+34.147454415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:11.934649 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:11.934634 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret podName:106281ea-1622-4966-bfd3-bdba5d770f5f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.934617967 +0000 UTC m=+34.147480153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret") pod "global-pull-secret-syncer-qkj2v" (UID: "106281ea-1622-4966-bfd3-bdba5d770f5f") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:12.136291 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:12.136272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:12.136393 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:12.136357 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:12.136393 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:12.136368 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:12.136393 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:12.136376 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xvx55 for pod openshift-network-diagnostics/network-check-target-6gwvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:12.136487 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:12.136413 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55 podName:b31374f9-c1aa-4288-a5db-c88ec5b3f732 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:28.136403044 +0000 UTC m=+34.349265216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvx55" (UniqueName: "kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55") pod "network-check-target-6gwvc" (UID: "b31374f9-c1aa-4288-a5db-c88ec5b3f732") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:12.276935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:12.276868 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:12.277051 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:12.276970 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:12.503215 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:12.503184 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="932504f980adca498715687d429473dab65cec5074590dc186cce3f64d459bde" exitCode=0 Oct 30 23:58:12.503215 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:12.503216 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerDied","Data":"932504f980adca498715687d429473dab65cec5074590dc186cce3f64d459bde"} Oct 30 23:58:13.276386 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:13.276349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:13.276386 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:13.276387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:13.276560 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:13.276480 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:13.276617 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:13.276601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:13.508014 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:13.507992 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:13.508428 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:13.508337 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"638236d85a2cfc4ac4f58ef6bc67c69856d870933e2cd22b6300e0b903579ba2"} Oct 30 23:58:13.511325 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:13.511301 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" event={"ID":"fbea15b0-87b2-480b-9bb7-496f71990632","Type":"ContainerStarted","Data":"639895c2062d09094763d3e39ee683b8524f4d45d1d51ce357ee1e6feace3b01"} Oct 30 23:58:13.537453 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:13.537385 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2fpcf" podStartSLOduration=8.813514779 podStartE2EDuration="19.537374112s" totalImagesPullingTime="10.723859333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:13.535903651 +0000 UTC m=+19.748765845" watchObservedRunningTime="2026-10-30 23:58:13.537374112 +0000 UTC m=+19.750236305" Oct 30 23:58:14.276833 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:14.276806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:14.276965 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:14.276891 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:15.276172 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.276144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:15.276172 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.276174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:15.276661 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:15.276243 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:15.276661 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:15.276293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:15.518824 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.518772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:15.519253 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.519229 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"f276a10a7eb2b2db4e42d74747df007d18a3f636065ea58d022cfe0cb02184e5"} Oct 30 23:58:15.519579 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.519553 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:15.519670 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.519634 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:15.519724 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.519663 2578 scope.go:122] "RemoveContainer" containerID="8c1c07030f27bd62d0112873e0e92da03390264eccfc3fda7474754c53eaa70f" Oct 30 23:58:15.536571 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.536551 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:15.537204 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:15.537188 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:16.277131 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:16.277101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:16.277813 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:16.277216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:16.524430 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:16.524404 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:16.524860 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:16.524839 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" event={"ID":"3db5e05a-532e-4937-a241-3a995c5c43ae","Type":"ContainerStarted","Data":"a97ee7bec568d4ae7e673ce7cfbca590eb2f98b07178af77a9c12efcebeaf11f"} Oct 30 23:58:16.524926 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:16.524900 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:16.558718 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:16.558650 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" podStartSLOduration=8.163542684 podStartE2EDuration="22.558637467s" totalImagesPullingTime="14.395094783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:16.557363821 +0000 UTC m=+22.770226015" watchObservedRunningTime="2026-10-30 23:58:16.558637467 +0000 UTC m=+22.771499653" Oct 30 23:58:17.276606 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.276576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:17.276886 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:17.276745 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:17.276886 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.276811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:17.277071 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:17.276905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:17.285308 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.285286 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkj2v"] Oct 30 23:58:17.293512 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.293488 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dzqgd"] Oct 30 23:58:17.293633 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.293574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:17.293686 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:17.293657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:17.295935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.295884 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gwvc"] Oct 30 23:58:17.526716 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.526649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:17.526864 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.526726 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:17.526864 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:17.526649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:17.526864 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:17.526761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:17.527192 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:17.527168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:19.277246 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:19.277080 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:19.277648 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:19.277124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:19.277648 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:19.277317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:19.277648 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:19.277123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:19.277648 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:19.277435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:19.277648 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:19.277471 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:21.276430 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:21.276393 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:21.276430 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:21.276438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:21.276917 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:21.276460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:21.276917 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:21.276552 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:21.276917 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:21.276692 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:21.276917 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:21.276760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:23.276638 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.276611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:23.277034 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.276611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:23.277034 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.276744 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzqgd" podUID="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" Oct 30 23:58:23.277034 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.276617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:23.277034 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.276779 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gwvc" podUID="b31374f9-c1aa-4288-a5db-c88ec5b3f732" Oct 30 23:58:23.277034 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.276871 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkj2v" podUID="106281ea-1622-4966-bfd3-bdba5d770f5f" Oct 30 23:58:23.570039 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.570020 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeReady" Oct 30 23:58:23.570144 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.570130 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:58:23.606009 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.605986 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tz88x"] Oct 30 23:58:23.608596 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.608582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:23.613858 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.613835 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-52d97"] Oct 30 23:58:23.623381 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.623362 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z49dz"] Oct 30 23:58:23.623500 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.623479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:23.626081 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.626061 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-46klc"] Oct 30 23:58:23.626181 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.626163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:23.626447 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.626425 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.626447 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.626425 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.630907 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.630513 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59457478cb-rqvwg"] Oct 30 23:58:23.631126 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.631093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:23.635435 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.635415 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq"] Oct 30 23:58:23.635687 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.635667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.640234 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.639677 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc"] Oct 30 23:58:23.640848 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.640405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.641676 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.641654 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fwwrl\"" Oct 30 23:58:23.642240 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.642010 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 23:58:23.643471 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.643451 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f8b974b69-pvb5d"] Oct 30 23:58:23.643612 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.643566 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:23.649797 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.649760 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nnh9c"] Oct 30 23:58:23.649935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.649915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.653099 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.652961 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz"] Oct 30 23:58:23.654135 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.654118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.656043 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.655944 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc"] Oct 30 23:58:23.657059 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.657022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.660231 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.660210 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sm45j"] Oct 30 23:58:23.660497 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.660481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.664996 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.664433 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69cf5d76f-vwpgd"] Oct 30 23:58:23.665397 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.665378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.668438 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.668077 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc"] Oct 30 23:58:23.668438 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.668362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.674971 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.674950 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc"] Oct 30 23:58:23.677577 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.677556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.681889 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.681865 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tz88x"] Oct 30 23:58:23.681969 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.681912 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5bnkm"] Oct 30 23:58:23.682420 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.682401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:23.686753 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.686736 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-52d97"] Oct 30 23:58:23.686917 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.686903 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z49dz"] Oct 30 23:58:23.687037 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.687016 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-46klc"] Oct 30 23:58:23.687139 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.687048 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq"] Oct 30 23:58:23.687139 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.687068 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59457478cb-rqvwg"] Oct 30 23:58:23.687139 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.687090 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nl6l6"] Oct 30 23:58:23.688020 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.688001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.690886 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690867 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc"] Oct 30 23:58:23.690968 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690897 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc"] Oct 30 23:58:23.690968 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690917 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc"] Oct 30 23:58:23.690968 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690941 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69cf5d76f-vwpgd"] Oct 30 23:58:23.690968 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690958 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f8b974b69-pvb5d"] Oct 30 23:58:23.690968 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690971 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nnh9c"] Oct 30 23:58:23.691209 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.690984 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc"] Oct 30 23:58:23.691209 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.691000 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sm45j"] Oct 30 23:58:23.691209 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.691007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.691350 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.691012 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bnkm"] Oct 30 23:58:23.691350 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.691286 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nl6l6"] Oct 30 23:58:23.691350 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.691302 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz"] Oct 30 23:58:23.720635 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.720611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.720776 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.720654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99-kube-api-access-jgfc9\") pod \"kas-connection-checker-685895c7c9-tz88x\" (UID: \"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99\") " pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:23.720966 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.720926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.721037 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.720979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.721037 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.720997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.721107 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2f8fec0-412d-40ee-9d94-db3fbe04a651-nginx-conf\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:23.721107 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-service-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.721182 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxcg\" (UniqueName: \"kubernetes.io/projected/b470b4c4-501f-4d51-a04d-37f1f3d71c06-kube-api-access-ztxcg\") pod \"volume-data-source-validator-58c9f5f46d-px2hc\" (UID: \"b470b4c4-501f-4d51-a04d-37f1f3d71c06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:23.721182 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-default-certificate\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.721240 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.721297 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lcv\" (UniqueName: \"kubernetes.io/projected/3ed58e8f-b3ed-4c94-aebf-0676c621799d-kube-api-access-26lcv\") pod \"kas-connection-checker-685895c7c9-z49dz\" (UID: \"3ed58e8f-b3ed-4c94-aebf-0676c621799d\") " pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:23.721297 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-serving-cert\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.721394 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.721394 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-snapshots\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.721394 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:23.721394 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.721584 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.721584 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.722004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.721978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/beff8893-2560-45f8-a96d-bce1e51dcf80-serving-cert\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.722274 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvzk\" (UniqueName: \"kubernetes.io/projected/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-kube-api-access-trvzk\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.722274 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.722274 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-tmp\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.722274 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.722274 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.722274 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fxt\" (UniqueName: \"kubernetes.io/projected/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-kube-api-access-q9fxt\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-stats-auth\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kbt5\" (UniqueName: \"kubernetes.io/projected/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-kube-api-access-9kbt5\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-config\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kxc\" (UniqueName: \"kubernetes.io/projected/93866d6e-e48c-4e7f-823d-771497d4694b-kube-api-access-x8kxc\") pod \"kas-connection-checker-685895c7c9-52d97\" (UID: \"93866d6e-e48c-4e7f-823d-771497d4694b\") " pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzqk\" (UniqueName: \"kubernetes.io/projected/6df4d978-5bb3-48e6-a69f-519f0d85e465-kube-api-access-wjzqk\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.722598 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.722519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jjjd\" (UniqueName: \"kubernetes.io/projected/beff8893-2560-45f8-a96d-bce1e51dcf80-kube-api-access-5jjjd\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.730212 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730181 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 23:58:23.730351 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730256 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.730351 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730321 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.730523 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730438 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730713 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730814 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730885 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730955 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.730994 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731051 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731090 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731165 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731204 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731297 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731319 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731340 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731397 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731458 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731597 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731644 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.731732 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731713 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731740 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731762 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.731971 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.732028 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.732036 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.732061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:23.732547 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.732128 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:23.737512 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.737492 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 23:58:23.743883 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.743868 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 23:58:23.743954 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.743924 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 23:58:23.744023 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.743971 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:58:23.744023 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744011 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 23:58:23.744114 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744071 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:58:23.744114 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744071 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 23:58:23.744351 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744337 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 23:58:23.744431 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744412 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 23:58:23.744431 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744366 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 23:58:23.744550 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744376 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 23:58:23.744550 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744341 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 23:58:23.744550 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744434 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 23:58:23.744765 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744743 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 23:58:23.744765 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744757 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpqng\"" Oct 30 23:58:23.744942 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744823 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 23:58:23.744942 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744834 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 23:58:23.744942 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744850 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxfk8\"" Oct 30 23:58:23.744942 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744872 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 23:58:23.744942 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744910 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vmg6c\"" Oct 30 23:58:23.745105 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.744974 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 23:58:23.823158 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.823258 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fxt\" (UniqueName: \"kubernetes.io/projected/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-kube-api-access-q9fxt\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.823258 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-stats-auth\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.823258 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbt5\" (UniqueName: \"kubernetes.io/projected/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-kube-api-access-9kbt5\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.823258 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.823258 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-config\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kxc\" (UniqueName: \"kubernetes.io/projected/93866d6e-e48c-4e7f-823d-771497d4694b-kube-api-access-x8kxc\") pod \"kas-connection-checker-685895c7c9-52d97\" (UID: \"93866d6e-e48c-4e7f-823d-771497d4694b\") " pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzqk\" (UniqueName: \"kubernetes.io/projected/6df4d978-5bb3-48e6-a69f-519f0d85e465-kube-api-access-wjzqk\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.823490 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjjd\" (UniqueName: \"kubernetes.io/projected/beff8893-2560-45f8-a96d-bce1e51dcf80-kube-api-access-5jjjd\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.823502 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.823504 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59457478cb-rqvwg: secret "image-registry-tls" not found Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6afa2442-496b-45d4-ae52-133e0d21a60f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.823554 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls podName:10737b2e-c98b-433d-a846-c087bfc5410f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.323535674 +0000 UTC m=+30.536397850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls") pod "image-registry-59457478cb-rqvwg" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f") : secret "image-registry-tls" not found Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa50726-3415-4c7f-82ba-e6eb5115267a-serving-cert\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c-kube-api-access-vdl2z\") pod \"network-check-source-6f78d59f99-mc2zc\" (UID: \"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.823683 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.32366428 +0000 UTC m=+30.536526467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99-kube-api-access-jgfc9\") pod \"kas-connection-checker-685895c7c9-tz88x\" (UID: \"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99\") " pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:23.823857 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-config\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823943 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2f8fec0-412d-40ee-9d94-db3fbe04a651-nginx-conf\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.823987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-service-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxcg\" (UniqueName: \"kubernetes.io/projected/b470b4c4-501f-4d51-a04d-37f1f3d71c06-kube-api-access-ztxcg\") pod \"volume-data-source-validator-58c9f5f46d-px2hc\" (UID: \"b470b4c4-501f-4d51-a04d-37f1f3d71c06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.824113 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwwx\" (UniqueName: \"kubernetes.io/projected/f270b734-9691-480f-8d55-1448c307a69b-kube-api-access-8fwwx\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.824158 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls podName:7dbbbfcf-c966-49b9-8cf3-91b77469bd2b nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.32414393 +0000 UTC m=+30.537006119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-whqjq" (UID: "7dbbbfcf-c966-49b9-8cf3-91b77469bd2b") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:23.824241 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-default-certificate\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.824707 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.824707 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.824707 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2f8fec0-412d-40ee-9d94-db3fbe04a651-nginx-conf\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:23.824707 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lcv\" (UniqueName: \"kubernetes.io/projected/3ed58e8f-b3ed-4c94-aebf-0676c621799d-kube-api-access-26lcv\") pod \"kas-connection-checker-685895c7c9-z49dz\" (UID: \"3ed58e8f-b3ed-4c94-aebf-0676c621799d\") " pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:23.824881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.824881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/6afa2442-496b-45d4-ae52-133e0d21a60f-kube-api-access-wg79k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.824881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27ea3-18d4-4aff-a92b-a4290ec72714-config-volume\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.824881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkh6\" (UniqueName: \"kubernetes.io/projected/e1a27ea3-18d4-4aff-a92b-a4290ec72714-kube-api-access-6jkh6\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.824881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.824881 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-serving-cert\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.825114 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-config\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.825114 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.825114 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.824995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-snapshots\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.825114 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.825027 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:58:23.825114 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f270b734-9691-480f-8d55-1448c307a69b-cert\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825159 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-service-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.825215 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.325202989 +0000 UTC m=+30.538065176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : secret "router-metrics-certs-default" not found Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.825387 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.825423 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.325411837 +0000 UTC m=+30.538274022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.825471 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a27ea3-18d4-4aff-a92b-a4290ec72714-tmp-dir\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-snapshots\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.825662 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/beff8893-2560-45f8-a96d-bce1e51dcf80-serving-cert\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.825698 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls podName:6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.325685764 +0000 UTC m=+30.538547964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pz8mc" (UID: "6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad") : secret "samples-operator-tls" not found Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa2442-496b-45d4-ae52-133e0d21a60f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.825825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5tn\" (UniqueName: \"kubernetes.io/projected/5fa50726-3415-4c7f-82ba-e6eb5115267a-kube-api-access-vq5tn\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvzk\" (UniqueName: \"kubernetes.io/projected/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-kube-api-access-trvzk\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-tmp\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.825999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.826039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.826081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.826196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.826296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.826347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.826330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-tmp\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.826997 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.826974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.827931 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.827912 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-serving-cert\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.828166 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.828145 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-stats-auth\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.828247 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.828233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.828398 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.828377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.828913 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.828897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-default-certificate\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.828913 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.828905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/beff8893-2560-45f8-a96d-bce1e51dcf80-serving-cert\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.837104 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.837082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fxt\" (UniqueName: \"kubernetes.io/projected/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-kube-api-access-q9fxt\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:23.837379 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.837361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99-kube-api-access-jgfc9\") pod \"kas-connection-checker-685895c7c9-tz88x\" (UID: \"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99\") " pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:23.837413 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.837403 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbt5\" (UniqueName: \"kubernetes.io/projected/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-kube-api-access-9kbt5\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:23.844702 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.844675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjjd\" (UniqueName: \"kubernetes.io/projected/beff8893-2560-45f8-a96d-bce1e51dcf80-kube-api-access-5jjjd\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:23.849645 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.849619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.849801 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.849730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:23.849999 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.849982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvzk\" (UniqueName: \"kubernetes.io/projected/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-kube-api-access-trvzk\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:23.850329 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.850305 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kxc\" (UniqueName: \"kubernetes.io/projected/93866d6e-e48c-4e7f-823d-771497d4694b-kube-api-access-x8kxc\") pod \"kas-connection-checker-685895c7c9-52d97\" (UID: \"93866d6e-e48c-4e7f-823d-771497d4694b\") " pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:23.850648 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.850630 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lcv\" (UniqueName: \"kubernetes.io/projected/3ed58e8f-b3ed-4c94-aebf-0676c621799d-kube-api-access-26lcv\") pod \"kas-connection-checker-685895c7c9-z49dz\" (UID: \"3ed58e8f-b3ed-4c94-aebf-0676c621799d\") " pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:23.850684 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.850644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzqk\" (UniqueName: \"kubernetes.io/projected/6df4d978-5bb3-48e6-a69f-519f0d85e465-kube-api-access-wjzqk\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:23.851549 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.851529 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxcg\" (UniqueName: \"kubernetes.io/projected/b470b4c4-501f-4d51-a04d-37f1f3d71c06-kube-api-access-ztxcg\") pod \"volume-data-source-validator-58c9f5f46d-px2hc\" (UID: \"b470b4c4-501f-4d51-a04d-37f1f3d71c06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:23.923825 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.923801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:23.927111 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa2442-496b-45d4-ae52-133e0d21a60f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.927195 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5tn\" (UniqueName: \"kubernetes.io/projected/5fa50726-3415-4c7f-82ba-e6eb5115267a-kube-api-access-vq5tn\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.927195 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.927195 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.927195 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.927355 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.927355 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.927248 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:23.927355 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.927265 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69cf5d76f-vwpgd: secret "image-registry-tls" not found Oct 30 23:58:23.927355 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.927312 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls podName:500001f6-9b33-4347-b5f5-cfcf79e6ed09 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.427295035 +0000 UTC m=+30.640157230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls") pod "image-registry-69cf5d76f-vwpgd" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09") : secret "image-registry-tls" not found Oct 30 23:58:23.927521 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6afa2442-496b-45d4-ae52-133e0d21a60f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.927521 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.927521 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa50726-3415-4c7f-82ba-e6eb5115267a-serving-cert\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.927521 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c-kube-api-access-vdl2z\") pod \"network-check-source-6f78d59f99-mc2zc\" (UID: \"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:23.927693 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwwx\" (UniqueName: \"kubernetes.io/projected/f270b734-9691-480f-8d55-1448c307a69b-kube-api-access-8fwwx\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.927693 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.927693 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/6afa2442-496b-45d4-ae52-133e0d21a60f-kube-api-access-wg79k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.927693 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27ea3-18d4-4aff-a92b-a4290ec72714-config-volume\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.927693 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkh6\" (UniqueName: \"kubernetes.io/projected/e1a27ea3-18d4-4aff-a92b-a4290ec72714-kube-api-access-6jkh6\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.927935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.927935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-config\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.927935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f270b734-9691-480f-8d55-1448c307a69b-cert\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.927935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.927935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a27ea3-18d4-4aff-a92b-a4290ec72714-tmp-dir\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.927935 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.928256 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.927952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.928256 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.928012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6afa2442-496b-45d4-ae52-133e0d21a60f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.928256 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.928108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.928406 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.928326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.928406 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.928328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.928941 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.928601 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:58:23.928941 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.928619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a27ea3-18d4-4aff-a92b-a4290ec72714-tmp-dir\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.928941 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:23.928651 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls podName:e1a27ea3-18d4-4aff-a92b-a4290ec72714 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:24.428636514 +0000 UTC m=+30.641498703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls") pod "dns-default-5bnkm" (UID: "e1a27ea3-18d4-4aff-a92b-a4290ec72714") : secret "dns-default-metrics-tls" not found Oct 30 23:58:23.929310 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.929262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.929310 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.929264 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27ea3-18d4-4aff-a92b-a4290ec72714-config-volume\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.929669 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.929340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-config\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.929919 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.929860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.930534 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.930483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa50726-3415-4c7f-82ba-e6eb5115267a-serving-cert\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.931032 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.931012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f270b734-9691-480f-8d55-1448c307a69b-cert\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.931640 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.931618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.931929 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.931906 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa2442-496b-45d4-ae52-133e0d21a60f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.942629 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.942609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:23.943170 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.943145 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.943516 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.943494 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5tn\" (UniqueName: \"kubernetes.io/projected/5fa50726-3415-4c7f-82ba-e6eb5115267a-kube-api-access-vq5tn\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:23.948501 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.948478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkh6\" (UniqueName: \"kubernetes.io/projected/e1a27ea3-18d4-4aff-a92b-a4290ec72714-kube-api-access-6jkh6\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:23.948981 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.948951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c-kube-api-access-vdl2z\") pod \"network-check-source-6f78d59f99-mc2zc\" (UID: \"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:23.949347 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.949309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/6afa2442-496b-45d4-ae52-133e0d21a60f-kube-api-access-wg79k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:23.949561 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.949543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:23.950533 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.950514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwwx\" (UniqueName: \"kubernetes.io/projected/f270b734-9691-480f-8d55-1448c307a69b-kube-api-access-8fwwx\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:23.955308 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.955243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:23.995070 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:23.995002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:24.007176 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.007081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:24.015192 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.015158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:24.032529 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.025909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:24.047452 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.047426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:24.051073 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.051007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:24.062343 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.062114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:24.219647 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.219621 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tz88x"] Oct 30 23:58:24.223176 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:24.223146 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da2a0da_b9b7_41f0_8e8d_6e07faa06d99.slice/crio-b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620 WatchSource:0}: Error finding container b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620: Status 404 returned error can't find the container with id b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620 Oct 30 23:58:24.268649 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.268594 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-52d97"] Oct 30 23:58:24.293847 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.292008 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z49dz"] Oct 30 23:58:24.307124 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:24.305889 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed58e8f_b3ed_4c94_aebf_0676c621799d.slice/crio-dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3 WatchSource:0}: Error finding container dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3: Status 404 returned error can't find the container with id dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3 Oct 30 23:58:24.343485 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.343457 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-nnh9c"] Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:24.345501 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeff8893_2560_45f8_a96d_bce1e51dcf80.slice/crio-b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79 WatchSource:0}: Error finding container b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79: Status 404 returned error can't find the container with id b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79 Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.346008 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.346605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.346665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.346709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.346796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.346826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.346911 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.346883056 +0000 UTC m=+31.559745235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : secret "router-metrics-certs-default" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.346938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347081 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347122 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls podName:7dbbbfcf-c966-49b9-8cf3-91b77469bd2b nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.347110349 +0000 UTC m=+31.559972535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-whqjq" (UID: "7dbbbfcf-c966-49b9-8cf3-91b77469bd2b") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347183 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347193 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59457478cb-rqvwg: secret "image-registry-tls" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347222 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls podName:10737b2e-c98b-433d-a846-c087bfc5410f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.34721215 +0000 UTC m=+31.560074323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls") pod "image-registry-59457478cb-rqvwg" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f") : secret "image-registry-tls" not found Oct 30 23:58:24.347921 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347277 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:58:24.348887 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347310 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls podName:6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.347299943 +0000 UTC m=+31.560162144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pz8mc" (UID: "6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad") : secret "samples-operator-tls" not found Oct 30 23:58:24.348887 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347370 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.347360371 +0000 UTC m=+31.560222577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:24.348887 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347393 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:24.348887 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.347449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.347431617 +0000 UTC m=+31.560293804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:24.348887 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.347640 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz"] Oct 30 23:58:24.357394 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:24.357372 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2587ab9_e842_47c6_87b5_fb0b1ab8b76e.slice/crio-3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f WatchSource:0}: Error finding container 3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f: Status 404 returned error can't find the container with id 3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f Oct 30 23:58:24.395165 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.395091 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc"] Oct 30 23:58:24.396228 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.396204 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sm45j"] Oct 30 23:58:24.398492 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.398407 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc"] Oct 30 23:58:24.401819 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.401770 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc"] Oct 30 23:58:24.401923 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.401848 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nl6l6"] Oct 30 23:58:24.447826 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.447803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:24.447918 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.447844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:24.447963 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.447950 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:58:24.448013 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.448004 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls podName:e1a27ea3-18d4-4aff-a92b-a4290ec72714 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.44798966 +0000 UTC m=+31.660851846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls") pod "dns-default-5bnkm" (UID: "e1a27ea3-18d4-4aff-a92b-a4290ec72714") : secret "dns-default-metrics-tls" not found Oct 30 23:58:24.448055 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.448008 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:24.448055 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.448024 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69cf5d76f-vwpgd: secret "image-registry-tls" not found Oct 30 23:58:24.448112 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:24.448059 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls podName:500001f6-9b33-4347-b5f5-cfcf79e6ed09 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.448047493 +0000 UTC m=+31.660909665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls") pod "image-registry-69cf5d76f-vwpgd" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09") : secret "image-registry-tls" not found Oct 30 23:58:24.542368 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.542310 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z49dz" event={"ID":"3ed58e8f-b3ed-4c94-aebf-0676c621799d","Type":"ContainerStarted","Data":"d914b27d86f920c8b8cd9ecc1d88937eaff02fe3c79ab6c9d8d714c9c9f6e199"} Oct 30 23:58:24.542368 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.542352 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z49dz" event={"ID":"3ed58e8f-b3ed-4c94-aebf-0676c621799d","Type":"ContainerStarted","Data":"dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3"} Oct 30 23:58:24.543501 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.543479 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" event={"ID":"6afa2442-496b-45d4-ae52-133e0d21a60f","Type":"ContainerStarted","Data":"63e174cc6e540a643b1b0660a64b97d1552042ec2d560daac796d571fda053a8"} Oct 30 23:58:24.544485 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.544467 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" event={"ID":"5fa50726-3415-4c7f-82ba-e6eb5115267a","Type":"ContainerStarted","Data":"a7e017f89a6abe46ebe21ba510dd13018c20c01ecf83cd8717169b6e2ef3335b"} Oct 30 23:58:24.545526 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.545498 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" event={"ID":"b470b4c4-501f-4d51-a04d-37f1f3d71c06","Type":"ContainerStarted","Data":"2fbe0f08468e86c474a57ebc457cab8c65e559ca14552f5e1ffb2cc1a86a26a4"} Oct 30 23:58:24.546534 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.546513 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" event={"ID":"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c","Type":"ContainerStarted","Data":"b1adfb9f1b0640f1c50c36b6969ee19eeec796889773ba3a26c76f30798c53ae"} Oct 30 23:58:24.547583 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.547563 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nl6l6" event={"ID":"f270b734-9691-480f-8d55-1448c307a69b","Type":"ContainerStarted","Data":"2c056bcec79eda73b369b99b7acc13686a621bf96b684c95d2f904da8a7db085"} Oct 30 23:58:24.548697 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.548675 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" event={"ID":"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e","Type":"ContainerStarted","Data":"3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f"} Oct 30 23:58:24.549716 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.549698 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" event={"ID":"beff8893-2560-45f8-a96d-bce1e51dcf80","Type":"ContainerStarted","Data":"b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79"} Oct 30 23:58:24.551082 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.551062 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-52d97" event={"ID":"93866d6e-e48c-4e7f-823d-771497d4694b","Type":"ContainerStarted","Data":"35c31ed7fa088fd9eb6ddad9c86698cadac42a1359ef94673f42fd811c99d305"} Oct 30 23:58:24.551173 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.551100 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-52d97" event={"ID":"93866d6e-e48c-4e7f-823d-771497d4694b","Type":"ContainerStarted","Data":"07d01ef2c2a8a18542a45e435249cfc41df999ff1b1d2496ec9ba336ce6ca08b"} Oct 30 23:58:24.552403 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.552385 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tz88x" event={"ID":"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99","Type":"ContainerStarted","Data":"fdf8b636da09c6e1af6b8821807d59a66a875537ea7bc12204805265990f2857"} Oct 30 23:58:24.552498 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.552415 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tz88x" event={"ID":"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99","Type":"ContainerStarted","Data":"b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620"} Oct 30 23:58:24.563581 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.563542 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-z49dz" podStartSLOduration=298.563532103 podStartE2EDuration="4m58.563532103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:24.562883174 +0000 UTC m=+30.775745369" watchObservedRunningTime="2026-10-30 23:58:24.563532103 +0000 UTC m=+30.776394296" Oct 30 23:58:24.595136 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.595019 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tz88x" podStartSLOduration=298.595008854 podStartE2EDuration="4m58.595008854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:24.594763772 +0000 UTC m=+30.807625965" watchObservedRunningTime="2026-10-30 23:58:24.595008854 +0000 UTC m=+30.807871049" Oct 30 23:58:24.615046 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:24.615010 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-52d97" podStartSLOduration=298.614995956 podStartE2EDuration="4m58.614995956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:24.614607394 +0000 UTC m=+30.827469587" watchObservedRunningTime="2026-10-30 23:58:24.614995956 +0000 UTC m=+30.827858150" Oct 30 23:58:25.278753 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.276655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:25.278753 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.277133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:25.278753 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.277567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:25.282147 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.282122 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:58:25.282750 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.282731 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:58:25.282963 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.282877 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:58:25.283155 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.283128 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:58:25.283229 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.283183 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:58:25.355987 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.355948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:25.356390 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.355996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:25.356390 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.356049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:25.356390 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.356117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:25.356390 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.356149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:25.356390 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.356213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:25.356390 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.356353 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:25.356732 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.356412 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls podName:7dbbbfcf-c966-49b9-8cf3-91b77469bd2b nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.35639439 +0000 UTC m=+33.569256565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-whqjq" (UID: "7dbbbfcf-c966-49b9-8cf3-91b77469bd2b") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:25.356864 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.356848 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:58:25.356937 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.356914 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.35689868 +0000 UTC m=+33.569760867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : secret "router-metrics-certs-default" not found Oct 30 23:58:25.356992 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.356986 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:25.357044 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357027 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.357008291 +0000 UTC m=+33.569870467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:25.357102 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357079 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:58:25.357152 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357118 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls podName:6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.357107529 +0000 UTC m=+33.569969715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pz8mc" (UID: "6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad") : secret "samples-operator-tls" not found Oct 30 23:58:25.357202 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357188 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:25.357202 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357199 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59457478cb-rqvwg: secret "image-registry-tls" not found Oct 30 23:58:25.357305 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357226 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls podName:10737b2e-c98b-433d-a846-c087bfc5410f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.357217246 +0000 UTC m=+33.570079422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls") pod "image-registry-59457478cb-rqvwg" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f") : secret "image-registry-tls" not found Oct 30 23:58:25.357305 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.357297 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.357288877 +0000 UTC m=+33.570151062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:25.457860 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.457829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:25.458017 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:25.457894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:25.458102 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.458063 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:25.458102 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.458079 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69cf5d76f-vwpgd: secret "image-registry-tls" not found Oct 30 23:58:25.458210 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.458143 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls podName:500001f6-9b33-4347-b5f5-cfcf79e6ed09 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.458117148 +0000 UTC m=+33.670979335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls") pod "image-registry-69cf5d76f-vwpgd" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09") : secret "image-registry-tls" not found Oct 30 23:58:25.458278 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.458225 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:58:25.458278 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:25.458275 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls podName:e1a27ea3-18d4-4aff-a92b-a4290ec72714 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:27.458259121 +0000 UTC m=+33.671121309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls") pod "dns-default-5bnkm" (UID: "e1a27ea3-18d4-4aff-a92b-a4290ec72714") : secret "dns-default-metrics-tls" not found Oct 30 23:58:26.408827 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:26.408221 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:26.408827 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:26.408471 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:26.433259 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:26.432909 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:27.378185 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.378147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:27.378463 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.378209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:27.378463 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.378250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:27.378463 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.378322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:27.378463 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.378357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:27.378463 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.378425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:27.378703 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.378563 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:27.378703 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.378628 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls podName:7dbbbfcf-c966-49b9-8cf3-91b77469bd2b nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.378609337 +0000 UTC m=+37.591471529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-whqjq" (UID: "7dbbbfcf-c966-49b9-8cf3-91b77469bd2b") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:27.379033 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379012 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:58:27.379270 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379068 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.379053477 +0000 UTC m=+37.591915662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : secret "router-metrics-certs-default" not found Oct 30 23:58:27.379270 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379129 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:27.379270 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379174 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.379153206 +0000 UTC m=+37.592015391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:27.379270 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379244 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:58:27.379637 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379280 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls podName:6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.379269601 +0000 UTC m=+37.592131787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pz8mc" (UID: "6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad") : secret "samples-operator-tls" not found Oct 30 23:58:27.379637 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379342 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:27.379637 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379355 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59457478cb-rqvwg: secret "image-registry-tls" not found Oct 30 23:58:27.379637 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379391 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls podName:10737b2e-c98b-433d-a846-c087bfc5410f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.379380625 +0000 UTC m=+37.592242816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls") pod "image-registry-59457478cb-rqvwg" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f") : secret "image-registry-tls" not found Oct 30 23:58:27.379637 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.379471 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.37945809 +0000 UTC m=+37.592320277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.479040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.479262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.479432 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.479496 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls podName:e1a27ea3-18d4-4aff-a92b-a4290ec72714 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.479477407 +0000 UTC m=+37.692339581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls") pod "dns-default-5bnkm" (UID: "e1a27ea3-18d4-4aff-a92b-a4290ec72714") : secret "dns-default-metrics-tls" not found Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.479898 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.479915 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69cf5d76f-vwpgd: secret "image-registry-tls" not found Oct 30 23:58:27.479994 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.479962 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls podName:500001f6-9b33-4347-b5f5-cfcf79e6ed09 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.479947126 +0000 UTC m=+37.692809322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls") pod "image-registry-69cf5d76f-vwpgd" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09") : secret "image-registry-tls" not found Oct 30 23:58:27.576111 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.575366 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" event={"ID":"b470b4c4-501f-4d51-a04d-37f1f3d71c06","Type":"ContainerStarted","Data":"261ebb1fb9db24c12d0cc841771667db7e5911ed5f1dee309cfd1556466334e0"} Oct 30 23:58:27.597830 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.596187 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" podStartSLOduration=272.203664962 podStartE2EDuration="4m34.596170557s" totalImagesPullingTime="2.392505595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:27.595749944 +0000 UTC m=+33.808612139" watchObservedRunningTime="2026-10-30 23:58:27.596170557 +0000 UTC m=+33.809032758" Oct 30 23:58:27.983695 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.983658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:27.983888 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.983720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:27.984360 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.984338 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:58:27.984458 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:27.984417 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:59.984396552 +0000 UTC m=+66.197258744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : secret "metrics-daemon-secret" not found Oct 30 23:58:27.987524 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:27.987497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:28.028470 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.028443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:28.185841 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.185366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:28.191968 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.191938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:28.324876 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.324765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:28.353218 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.353187 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkj2v"] Oct 30 23:58:28.363958 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:28.363927 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106281ea_1622_4966_bfd3_bdba5d770f5f.slice/crio-c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc WatchSource:0}: Error finding container c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc: Status 404 returned error can't find the container with id c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc Oct 30 23:58:28.580800 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.580682 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qkj2v" event={"ID":"106281ea-1622-4966-bfd3-bdba5d770f5f","Type":"ContainerStarted","Data":"c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc"} Oct 30 23:58:28.609966 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:28.609810 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gwvc"] Oct 30 23:58:28.699955 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:28.699924 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31374f9_c1aa_4288_a5db_c88ec5b3f732.slice/crio-31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3 WatchSource:0}: Error finding container 31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3: Status 404 returned error can't find the container with id 31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3 Oct 30 23:58:29.592882 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.592843 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" event={"ID":"6afa2442-496b-45d4-ae52-133e0d21a60f","Type":"ContainerStarted","Data":"2e2741d7dd465702f282be021274e55f4269f2f453ec8296a332791204090a82"} Oct 30 23:58:29.595134 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.595101 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/0.log" Oct 30 23:58:29.595278 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.595148 2578 generic.go:356] "Generic (PLEG): container finished" podID="5fa50726-3415-4c7f-82ba-e6eb5115267a" containerID="ab87a7ab8fc9b1a5058bb2abd2088eb1f7052fd31ab90ec67c397ea26dba3d05" exitCode=255 Oct 30 23:58:29.595278 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.595215 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" event={"ID":"5fa50726-3415-4c7f-82ba-e6eb5115267a","Type":"ContainerDied","Data":"ab87a7ab8fc9b1a5058bb2abd2088eb1f7052fd31ab90ec67c397ea26dba3d05"} Oct 30 23:58:29.595420 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.595383 2578 scope.go:122] "RemoveContainer" containerID="ab87a7ab8fc9b1a5058bb2abd2088eb1f7052fd31ab90ec67c397ea26dba3d05" Oct 30 23:58:29.596650 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.596618 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gwvc" event={"ID":"b31374f9-c1aa-4288-a5db-c88ec5b3f732","Type":"ContainerStarted","Data":"31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3"} Oct 30 23:58:29.637324 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.637273 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" podStartSLOduration=272.040626933 podStartE2EDuration="4m36.637256606s" totalImagesPullingTime="4.596629673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:29.613879218 +0000 UTC m=+35.826741413" watchObservedRunningTime="2026-10-30 23:58:29.637256606 +0000 UTC m=+35.850118800" Oct 30 23:58:29.975811 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.975367 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps"] Oct 30 23:58:29.989356 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.989296 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4"] Oct 30 23:58:29.991711 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.991023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:29.999047 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.998878 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5dm5b\"" Oct 30 23:58:29.999047 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.998961 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 23:58:29.999047 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:29.999008 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:30.003207 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.002973 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps"] Oct 30 23:58:30.003207 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.003011 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4"] Oct 30 23:58:30.003207 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.003095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:30.106646 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.106616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5l2\" (UniqueName: \"kubernetes.io/projected/d0ffac65-ad93-4617-afbd-9d2fe6153391-kube-api-access-wb5l2\") pod \"migrator-66df5d496b-wmdc4\" (UID: \"d0ffac65-ad93-4617-afbd-9d2fe6153391\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:30.106798 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.106739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq72r\" (UniqueName: \"kubernetes.io/projected/686b1996-b6f2-44e9-b9bd-8acb156e833d-kube-api-access-zq72r\") pod \"migrator-66df5d496b-jwqps\" (UID: \"686b1996-b6f2-44e9-b9bd-8acb156e833d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:30.208315 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.208278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq72r\" (UniqueName: \"kubernetes.io/projected/686b1996-b6f2-44e9-b9bd-8acb156e833d-kube-api-access-zq72r\") pod \"migrator-66df5d496b-jwqps\" (UID: \"686b1996-b6f2-44e9-b9bd-8acb156e833d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:30.208486 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.208409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5l2\" (UniqueName: \"kubernetes.io/projected/d0ffac65-ad93-4617-afbd-9d2fe6153391-kube-api-access-wb5l2\") pod \"migrator-66df5d496b-wmdc4\" (UID: \"d0ffac65-ad93-4617-afbd-9d2fe6153391\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:30.234975 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.234944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5l2\" (UniqueName: \"kubernetes.io/projected/d0ffac65-ad93-4617-afbd-9d2fe6153391-kube-api-access-wb5l2\") pod \"migrator-66df5d496b-wmdc4\" (UID: \"d0ffac65-ad93-4617-afbd-9d2fe6153391\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:30.239474 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.239422 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq72r\" (UniqueName: \"kubernetes.io/projected/686b1996-b6f2-44e9-b9bd-8acb156e833d-kube-api-access-zq72r\") pod \"migrator-66df5d496b-jwqps\" (UID: \"686b1996-b6f2-44e9-b9bd-8acb156e833d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:30.360287 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.360225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:30.375430 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.375376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:30.604896 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.604865 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 30 23:58:30.605682 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.605662 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/0.log" Oct 30 23:58:30.605805 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.605698 2578 generic.go:356] "Generic (PLEG): container finished" podID="5fa50726-3415-4c7f-82ba-e6eb5115267a" containerID="3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d" exitCode=255 Oct 30 23:58:30.605805 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.605768 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" event={"ID":"5fa50726-3415-4c7f-82ba-e6eb5115267a","Type":"ContainerDied","Data":"3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d"} Oct 30 23:58:30.605957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.605843 2578 scope.go:122] "RemoveContainer" containerID="ab87a7ab8fc9b1a5058bb2abd2088eb1f7052fd31ab90ec67c397ea26dba3d05" Oct 30 23:58:30.606286 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.606261 2578 scope.go:122] "RemoveContainer" containerID="3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d" Oct 30 23:58:30.606503 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:30.606481 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sm45j_openshift-console-operator(5fa50726-3415-4c7f-82ba-e6eb5115267a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" podUID="5fa50726-3415-4c7f-82ba-e6eb5115267a" Oct 30 23:58:30.609646 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.609598 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" event={"ID":"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c","Type":"ContainerStarted","Data":"70a2d9ca1fbd7db730f29ab93fae7bd071993876220d6552e2709faeada52ae4"} Oct 30 23:58:30.615067 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.615013 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nl6l6" event={"ID":"f270b734-9691-480f-8d55-1448c307a69b","Type":"ContainerStarted","Data":"a0e2bcdad9f0a8e232a05f25f161ce3d5e93d5ce136b9f455100ccfc9d81b527"} Oct 30 23:58:30.617886 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.617599 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" event={"ID":"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e","Type":"ContainerStarted","Data":"2057f0b41c2e1145df4c43efe708098fed2cd2168d2d4beacbca4f80ab50fb0b"} Oct 30 23:58:30.619575 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.619552 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" event={"ID":"beff8893-2560-45f8-a96d-bce1e51dcf80","Type":"ContainerStarted","Data":"f0c4a8ce8962645de603ea9cb8ce4c91dfaafedb5c1314b29f3c08e5cc5f3c7d"} Oct 30 23:58:30.654753 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.654703 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" podStartSLOduration=272.243865601 podStartE2EDuration="4m37.654687452s" totalImagesPullingTime="5.410821851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:30.653008907 +0000 UTC m=+36.865871101" watchObservedRunningTime="2026-10-30 23:58:30.654687452 +0000 UTC m=+36.867549647" Oct 30 23:58:30.675213 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.675152 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" podStartSLOduration=272.259973328 podStartE2EDuration="4m37.675136227s" totalImagesPullingTime="5.415162899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:30.67397247 +0000 UTC m=+36.886834667" watchObservedRunningTime="2026-10-30 23:58:30.675136227 +0000 UTC m=+36.887998422" Oct 30 23:58:30.676050 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.675374 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps"] Oct 30 23:58:30.717840 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.717772 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nl6l6" podStartSLOduration=2.37132332 podStartE2EDuration="7.717754614s" totalImagesPullingTime="5.346431294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:30.7117722 +0000 UTC m=+36.924634394" watchObservedRunningTime="2026-10-30 23:58:30.717754614 +0000 UTC m=+36.930616809" Oct 30 23:58:30.719010 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.718918 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4"] Oct 30 23:58:30.900565 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.900464 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" podStartSLOduration=269.233282845 podStartE2EDuration="4m34.900448239s" totalImagesPullingTime="5.667165394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:30.739074986 +0000 UTC m=+36.951937185" watchObservedRunningTime="2026-10-30 23:58:30.900448239 +0000 UTC m=+37.113310433" Oct 30 23:58:30.901672 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.901645 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-kxzdr"] Oct 30 23:58:30.934326 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.934300 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-kxzdr"] Oct 30 23:58:30.934486 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.934413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:30.940053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.939593 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 23:58:30.940053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.939851 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 23:58:30.940053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.939956 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 23:58:30.940053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.940013 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 23:58:30.940242 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:30.940094 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j47x2\"" Oct 30 23:58:31.006267 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.006238 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jvn2x"] Oct 30 23:58:31.024720 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.024695 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bbltk-htq8x"] Oct 30 23:58:31.024912 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.024886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.040354 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.040322 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvn2x"] Oct 30 23:58:31.040485 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.040373 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bbltk-htq8x"] Oct 30 23:58:31.040485 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.040451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.044312 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.044286 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:58:31.045282 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.045259 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:58:31.045579 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.045562 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7b2l\"" Oct 30 23:58:31.045657 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.045634 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6l954\"" Oct 30 23:58:31.120855 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.120834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-api-access-kcnkw\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.120977 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.120905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.120977 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.120926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-crio-socket\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.121053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.120992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.121053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.121014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-data-volume\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.121053 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.121039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2528eea-3e1e-49eb-ad02-537965818be1-signing-key\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.121151 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.121067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2528eea-3e1e-49eb-ad02-537965818be1-signing-cabundle\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.121151 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.121081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpfp\" (UniqueName: \"kubernetes.io/projected/b2528eea-3e1e-49eb-ad02-537965818be1-kube-api-access-prpfp\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.221486 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2528eea-3e1e-49eb-ad02-537965818be1-signing-key\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.221486 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2528eea-3e1e-49eb-ad02-537965818be1-signing-cabundle\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.221677 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpfp\" (UniqueName: \"kubernetes.io/projected/b2528eea-3e1e-49eb-ad02-537965818be1-kube-api-access-prpfp\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.221677 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-api-access-kcnkw\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.221830 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.221830 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-crio-socket\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.221830 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.221830 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.221816 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:31.221830 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.222074 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.221867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-crio-socket\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.222074 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.221905 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.721884143 +0000 UTC m=+37.934746316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:31.222074 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.222009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.222180 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.222070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.222180 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.222131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.222257 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.222184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-data-volume\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.222257 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.222242 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:58:31.222535 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.222520 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-data-volume\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.222639 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.222613 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:31.722276329 +0000 UTC m=+37.935138506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : configmap "kube-rbac-proxy" not found Oct 30 23:58:31.222913 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.222888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2528eea-3e1e-49eb-ad02-537965818be1-signing-cabundle\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.224957 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.224937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2528eea-3e1e-49eb-ad02-537965818be1-signing-key\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.243284 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.243260 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-api-access-kcnkw\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.243627 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.243579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpfp\" (UniqueName: \"kubernetes.io/projected/b2528eea-3e1e-49eb-ad02-537965818be1-kube-api-access-prpfp\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.260885 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.260859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:31.327460 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.327424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.327604 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.327486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.327604 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.327530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.327604 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.327571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.328548 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.328221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.328665 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.328582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.329127 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.329062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.342577 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.342542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.417004 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.416769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:31.428483 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.428458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:31.428639 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.428495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:31.428639 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.428558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:31.428639 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.428615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:31.428639 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428628 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428643 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59457478cb-rqvwg: secret "image-registry-tls" not found Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.428648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428699 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls podName:10737b2e-c98b-433d-a846-c087bfc5410f nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.428680112 +0000 UTC m=+45.641542304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls") pod "image-registry-59457478cb-rqvwg" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f") : secret "image-registry-tls" not found Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.428718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428739 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428773 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.428762325 +0000 UTC m=+45.641624515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:31.428838 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428833 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:58:31.429273 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428866 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls podName:6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.428855265 +0000 UTC m=+45.641717440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-pz8mc" (UID: "6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad") : secret "samples-operator-tls" not found Oct 30 23:58:31.429273 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428900 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.428891228 +0000 UTC m=+45.641753403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:31.429273 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428951 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:31.429273 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.428979 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls podName:7dbbbfcf-c966-49b9-8cf3-91b77469bd2b nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.428969711 +0000 UTC m=+45.641831885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-whqjq" (UID: "7dbbbfcf-c966-49b9-8cf3-91b77469bd2b") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:58:31.429273 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.429025 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:58:31.429273 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.429054 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.429043138 +0000 UTC m=+45.641905327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : secret "router-metrics-certs-default" not found Oct 30 23:58:31.529654 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.529619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:31.529806 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.529698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:31.529980 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.529962 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:58:31.530095 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.529985 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69cf5d76f-vwpgd: secret "image-registry-tls" not found Oct 30 23:58:31.530095 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.530041 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls podName:500001f6-9b33-4347-b5f5-cfcf79e6ed09 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.530023011 +0000 UTC m=+45.742885198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls") pod "image-registry-69cf5d76f-vwpgd" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09") : secret "image-registry-tls" not found Oct 30 23:58:31.530231 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.530117 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:58:31.530231 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.530154 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls podName:e1a27ea3-18d4-4aff-a92b-a4290ec72714 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:39.530142148 +0000 UTC m=+45.743004336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls") pod "dns-default-5bnkm" (UID: "e1a27ea3-18d4-4aff-a92b-a4290ec72714") : secret "dns-default-metrics-tls" not found Oct 30 23:58:31.575179 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.575087 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-kxzdr"] Oct 30 23:58:31.579698 ip-10-0-140-246 kubenswrapper[2578]: W1030 23:58:31.579662 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2528eea_3e1e_49eb_ad02_537965818be1.slice/crio-95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d WatchSource:0}: Error finding container 95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d: Status 404 returned error can't find the container with id 95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d Oct 30 23:58:31.626517 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.626493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 30 23:58:31.626991 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.626949 2578 scope.go:122] "RemoveContainer" containerID="3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d" Oct 30 23:58:31.627366 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.627160 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sm45j_openshift-console-operator(5fa50726-3415-4c7f-82ba-e6eb5115267a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" podUID="5fa50726-3415-4c7f-82ba-e6eb5115267a" Oct 30 23:58:31.630543 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.630479 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" event={"ID":"b2528eea-3e1e-49eb-ad02-537965818be1","Type":"ContainerStarted","Data":"95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d"} Oct 30 23:58:31.632225 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.632201 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" event={"ID":"d0ffac65-ad93-4617-afbd-9d2fe6153391","Type":"ContainerStarted","Data":"c1a5c70626fea5bf132fdf22a1d05d73abd4621b6bd713a12836d3a12f7e3c71"} Oct 30 23:58:31.634264 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.634239 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" event={"ID":"686b1996-b6f2-44e9-b9bd-8acb156e833d","Type":"ContainerStarted","Data":"667d3f69c6bfa2fb317ec78ed34c0fc8687c98a92aeaae0c70abab59cf2244d5"} Oct 30 23:58:31.731001 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.730967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.731140 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.731111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:31.732051 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.732027 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:31.732174 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.732095 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:32.732074929 +0000 UTC m=+38.944937120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:31.732174 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.732145 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:58:31.732312 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:31.732183 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:32.732171053 +0000 UTC m=+38.945033238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : configmap "kube-rbac-proxy" not found Oct 30 23:58:31.734775 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:31.734737 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bbltk-htq8x"] Oct 30 23:58:32.650947 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.648896 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gwvc" event={"ID":"b31374f9-c1aa-4288-a5db-c88ec5b3f732","Type":"ContainerStarted","Data":"2b4315b52e74127b85b3ac42f59dbc9edc417496b2869db6c4a330b8fc2f3164"} Oct 30 23:58:32.650947 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.648978 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:32.654702 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.654264 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" event={"ID":"b2528eea-3e1e-49eb-ad02-537965818be1","Type":"ContainerStarted","Data":"d82f1362abe70c906c6fdc3ff15b5a360f1eba5590fecb77ae008b89bbc1ce00"} Oct 30 23:58:32.657936 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.656765 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bbltk-htq8x" event={"ID":"d91a42d7-9685-4f8f-b620-5e93d58b1a76","Type":"ContainerStarted","Data":"30baadfeac507322a135f2667e34b9f22cdd354c91289a720aa1356301e3e0cf"} Oct 30 23:58:32.657936 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.656834 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bbltk-htq8x" event={"ID":"d91a42d7-9685-4f8f-b620-5e93d58b1a76","Type":"ContainerStarted","Data":"14678713d30ae56b8d949a1122f2428e6956a5d14af1a67d12a4fb48d42860b5"} Oct 30 23:58:32.672958 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.672902 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6gwvc" podStartSLOduration=35.381408342 podStartE2EDuration="38.672876141s" totalImagesPullingTime="3.291467799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:32.671999424 +0000 UTC m=+38.884861620" watchObservedRunningTime="2026-10-30 23:58:32.672876141 +0000 UTC m=+38.885738336" Oct 30 23:58:32.694179 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.694125 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bbltk-htq8x" podStartSLOduration=2.694107986 podStartE2EDuration="2.694107986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:32.692176079 +0000 UTC m=+38.905038274" watchObservedRunningTime="2026-10-30 23:58:32.694107986 +0000 UTC m=+38.906970184" Oct 30 23:58:32.715802 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.715731 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" podStartSLOduration=2.715709649 podStartE2EDuration="2.715709649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:32.71450955 +0000 UTC m=+38.927371743" watchObservedRunningTime="2026-10-30 23:58:32.715709649 +0000 UTC m=+38.928571843" Oct 30 23:58:32.745826 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.743500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:32.745826 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:32.743715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:32.745826 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:32.744307 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:58:32.745826 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:32.744369 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:34.744351181 +0000 UTC m=+40.957213370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : configmap "kube-rbac-proxy" not found Oct 30 23:58:32.745826 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:32.745664 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:32.745826 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:32.745711 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:34.745695799 +0000 UTC m=+40.958557988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:33.664430 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:33.664384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" event={"ID":"d0ffac65-ad93-4617-afbd-9d2fe6153391","Type":"ContainerStarted","Data":"b381443c3bce9b16364139d5207bbb3498bc43093e2b182cd0abfd1f587029f4"} Oct 30 23:58:33.667648 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:33.667617 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" event={"ID":"686b1996-b6f2-44e9-b9bd-8acb156e833d","Type":"ContainerStarted","Data":"ad012865ff8f1cd9ee21a52cd4022b461d41367fc2d82bccdfa0b6bec6186321"} Oct 30 23:58:33.683309 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:33.683258 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" podStartSLOduration=2.769367511 podStartE2EDuration="4.683240764s" totalImagesPullingTime="1.913873253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:33.682025682 +0000 UTC m=+39.894887883" watchObservedRunningTime="2026-10-30 23:58:33.683240764 +0000 UTC m=+39.896102959" Oct 30 23:58:33.708564 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:33.708506 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" podStartSLOduration=1.8539561469999999 podStartE2EDuration="4.708489716s" totalImagesPullingTime="2.854533569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:33.707684449 +0000 UTC m=+39.920546644" watchObservedRunningTime="2026-10-30 23:58:33.708489716 +0000 UTC m=+39.921351909" Oct 30 23:58:34.026294 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:34.026248 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:34.026468 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:34.026321 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:34.026738 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:34.026722 2578 scope.go:122] "RemoveContainer" containerID="3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d" Oct 30 23:58:34.026997 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:34.026971 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sm45j_openshift-console-operator(5fa50726-3415-4c7f-82ba-e6eb5115267a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" podUID="5fa50726-3415-4c7f-82ba-e6eb5115267a" Oct 30 23:58:34.033603 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:34.033582 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjnpt_b613d66f-e947-427e-9694-3f0b609f35c4/dns-node-resolver/0.log" Oct 30 23:58:34.761040 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:34.761005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:34.761499 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:34.761127 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:58:34.761499 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:34.761201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:38.761183834 +0000 UTC m=+44.974046030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : configmap "kube-rbac-proxy" not found Oct 30 23:58:34.761499 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:34.761245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:34.761499 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:34.761347 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:34.761499 ip-10-0-140-246 kubenswrapper[2578]: E1030 23:58:34.761380 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:38.761371519 +0000 UTC m=+44.974233704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:35.034231 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:35.034153 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n28vw_71c479d0-cb02-4549-acb7-3a206eb2ecb7/node-ca/0.log" Oct 30 23:58:35.633384 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:35.633363 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nl6l6_f270b734-9691-480f-8d55-1448c307a69b/serve-healthcheck-canary/0.log" Oct 30 23:58:35.830721 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:35.830645 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jwqps_686b1996-b6f2-44e9-b9bd-8acb156e833d/migrator/0.log" Oct 30 23:58:35.886763 ip-10-0-140-246 kubenswrapper[2578]: I1030 23:58:35.886732 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:35.886929 ip-10-0-140-246 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:58:35.899118 ip-10-0-140-246 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:58:35.899293 ip-10-0-140-246 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:58:35.899339 ip-10-0-140-246 systemd[1]: kubelet.service: Consumed 3.177s CPU time. Oct 30 23:58:35.908504 ip-10-0-140-246 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:58:35.979920 ip-10-0-140-246 kubenswrapper[5073]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:35.979920 ip-10-0-140-246 kubenswrapper[5073]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:58:35.979920 ip-10-0-140-246 kubenswrapper[5073]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:35.979920 ip-10-0-140-246 kubenswrapper[5073]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:35.981957 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981885 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:35.981957 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981956 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:35.981957 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981960 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981963 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981966 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981969 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981971 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981974 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981978 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981983 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981986 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981989 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981992 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981995 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.981997 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982000 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982002 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982005 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982007 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982010 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982012 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:35.982075 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982015 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982018 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982021 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982024 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982026 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982029 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982039 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982042 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982045 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982047 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982050 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982053 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982055 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982058 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982060 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982063 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982065 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982068 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982070 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982073 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:35.982563 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982076 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982079 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982081 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982084 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982087 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982089 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982092 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982095 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982098 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982100 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982103 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982105 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982108 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982110 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982113 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982117 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982121 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982124 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982127 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:35.983110 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982130 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982133 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982135 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982139 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982142 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982147 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982150 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982152 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982155 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982158 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982161 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982163 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982166 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982168 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982171 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982173 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982176 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982178 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982181 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982184 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:35.983559 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982187 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982190 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982192 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982194 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982204 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982210 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982213 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982215 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982218 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982221 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982223 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982227 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982230 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982233 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982235 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982237 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982240 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982242 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982245 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982247 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:35.984073 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982250 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982253 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982255 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982258 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982260 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.982263 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983212 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983220 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983224 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983227 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983230 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983235 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983238 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983241 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983244 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983246 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983249 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983254 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983256 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:35.984552 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983259 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983261 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983264 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983267 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983269 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983272 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983274 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983277 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983279 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983282 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983285 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983287 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983290 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983292 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983296 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983298 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983301 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983303 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983306 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983308 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:35.985028 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983311 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983313 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983316 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983318 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983321 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983324 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983326 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983329 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983331 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983334 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983337 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983340 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983342 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983345 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983347 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983350 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983352 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983355 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983357 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983360 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:35.985511 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983362 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983364 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983367 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983369 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983372 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983374 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983379 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983383 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983385 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983388 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983391 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983393 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983396 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983398 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983401 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983403 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983406 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983408 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983411 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:35.986024 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983414 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983416 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983419 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983423 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983425 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983428 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983430 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983433 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983435 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983438 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983440 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983443 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983445 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983448 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983450 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983453 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983455 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983458 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983460 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983463 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:35.986491 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983465 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983467 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983470 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983472 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983474 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983477 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983479 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983483 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983487 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983490 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983493 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983495 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983503 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.983510 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984015 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984023 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984026 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984029 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984032 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:35.987000 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984035 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984037 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984040 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984043 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984046 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984048 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984051 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984054 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984056 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984059 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984062 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984064 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984066 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984069 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984072 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984074 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984077 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984080 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984083 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:35.987469 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984085 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984088 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984091 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984093 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984096 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984099 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984102 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984104 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984107 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984110 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984113 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984115 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984117 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984120 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984122 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984125 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984127 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984129 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984132 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984135 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:35.988085 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984137 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984140 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984142 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984144 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984147 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984149 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984152 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984154 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984157 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984159 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984161 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984166 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984170 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984173 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984175 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984178 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984180 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984183 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984185 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:35.988606 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984189 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984191 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984194 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984196 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984199 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984201 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984204 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984206 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984208 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984211 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984213 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984216 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984218 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984220 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984239 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984243 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984246 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984248 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984251 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984253 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:35.989082 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984256 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984259 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984261 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984264 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984266 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984269 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984272 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984274 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984277 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984279 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984281 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984284 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984287 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984295 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984298 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984301 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984303 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984306 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984309 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984312 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:35.989564 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984314 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984316 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.984319 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984410 5073 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984421 5073 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984427 5073 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984431 5073 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984436 5073 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984439 5073 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984443 5073 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984447 5073 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984450 5073 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984453 5073 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984457 5073 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984460 5073 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984463 5073 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984466 5073 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984469 5073 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984472 5073 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984475 5073 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984478 5073 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984481 5073 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984484 5073 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:58:35.990087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984487 5073 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984490 5073 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984494 5073 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984498 5073 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984501 5073 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984504 5073 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984507 5073 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984510 5073 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984513 5073 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984516 5073 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984519 5073 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984522 5073 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984526 5073 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984529 5073 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984532 5073 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984534 5073 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984537 5073 flags.go:64] FLAG: --enable-server="true" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984540 5073 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984544 5073 flags.go:64] FLAG: --event-burst="100" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984547 5073 flags.go:64] FLAG: --event-qps="50" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984552 5073 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984555 5073 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984558 5073 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984562 5073 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:58:35.990644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984565 5073 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984568 5073 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984571 5073 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984574 5073 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984576 5073 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984579 5073 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984582 5073 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984585 5073 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984587 5073 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984590 5073 flags.go:64] FLAG: --feature-gates="" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984594 5073 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984597 5073 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984609 5073 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984612 5073 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984615 5073 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984619 5073 flags.go:64] FLAG: --help="false" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984621 5073 flags.go:64] FLAG: --hostname-override="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984624 5073 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984627 5073 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984630 5073 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984634 5073 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984637 5073 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984640 5073 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:58:35.991244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984643 5073 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984645 5073 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984649 5073 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984651 5073 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984654 5073 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984658 5073 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984661 5073 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984665 5073 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984668 5073 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984671 5073 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984673 5073 flags.go:64] FLAG: --lock-file="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984676 5073 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984679 5073 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984682 5073 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984687 5073 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984690 5073 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984693 5073 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984696 5073 flags.go:64] FLAG: --logging-format="text" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984699 5073 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984702 5073 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984705 5073 flags.go:64] FLAG: --manifest-url="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984709 5073 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984714 5073 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984717 5073 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984720 5073 flags.go:64] FLAG: --max-pods="110" Oct 30 23:58:35.991814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984723 5073 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984726 5073 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984729 5073 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984732 5073 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984736 5073 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984738 5073 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984741 5073 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984748 5073 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984751 5073 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984754 5073 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984757 5073 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984760 5073 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984765 5073 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984768 5073 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984771 5073 flags.go:64] FLAG: --port="10250" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984774 5073 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984777 5073 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02422223eb39a111c" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984794 5073 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984797 5073 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984800 5073 flags.go:64] FLAG: --register-node="true" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984803 5073 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984807 5073 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984810 5073 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984813 5073 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984815 5073 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984819 5073 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:58:35.992445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984822 5073 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984825 5073 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984829 5073 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984832 5073 flags.go:64] FLAG: --runonce="false" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984841 5073 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984844 5073 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984847 5073 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984850 5073 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984853 5073 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984856 5073 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984859 5073 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984861 5073 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984864 5073 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984867 5073 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984870 5073 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984873 5073 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984876 5073 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984878 5073 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984883 5073 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984888 5073 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984890 5073 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984893 5073 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984901 5073 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984904 5073 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984907 5073 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:58:35.993127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984909 5073 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984912 5073 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984915 5073 flags.go:64] FLAG: --v="2" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984919 5073 flags.go:64] FLAG: --version="false" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984923 5073 flags.go:64] FLAG: --vmodule="" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984927 5073 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.984930 5073 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985076 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985080 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985084 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985087 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985089 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985094 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985097 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985100 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985103 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985105 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985108 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985110 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985113 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985116 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:35.993984 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985118 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985121 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985123 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985126 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985130 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985132 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985135 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985137 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985140 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985143 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985145 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985148 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985150 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985153 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985155 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985157 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985160 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985162 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985165 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:35.994845 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985167 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985172 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985174 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985177 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985180 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985183 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985185 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985188 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985190 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985192 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985195 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985198 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985200 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985202 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985205 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985207 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985210 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985214 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985216 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985219 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:35.995365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985221 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985224 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985226 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985229 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985231 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985234 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985236 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985239 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985241 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985244 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985246 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985248 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985252 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985258 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985261 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985264 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985267 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985270 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985273 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:35.996034 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985275 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985278 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985280 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985283 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985285 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985288 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985290 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985293 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985295 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985298 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985301 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985304 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985307 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985309 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985312 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985314 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985317 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985319 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985322 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985324 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:35.996528 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985327 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985329 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985332 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985334 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985337 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985339 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985343 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985346 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985348 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985351 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985354 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985356 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985359 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985361 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.985387 5073 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985421 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:35.997121 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985425 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985428 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985430 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985433 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985436 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985439 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985443 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985446 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985448 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985451 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985453 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985456 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985458 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985461 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985463 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985466 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985468 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985471 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985474 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:35.997529 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985476 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985478 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985481 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985483 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985487 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985490 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985492 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985495 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985498 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985502 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985505 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985508 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985511 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985513 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985516 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985518 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985521 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985524 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985526 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:35.998012 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985530 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985533 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985535 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985538 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985540 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985543 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985545 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985548 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985550 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985553 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985556 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985559 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985561 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985563 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985566 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985568 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985571 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985575 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985578 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985580 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:35.998590 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985583 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985585 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985587 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985590 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985593 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985595 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985598 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985601 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985603 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985605 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985608 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985610 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985614 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985616 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985619 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985622 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985626 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985628 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985631 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985634 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:35.999231 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985636 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985638 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985641 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985643 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985646 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985648 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985650 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985653 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985655 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985659 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985662 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985664 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985667 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985670 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985673 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985675 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985678 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985681 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985683 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:35.999762 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985686 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985688 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985691 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985693 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985695 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985699 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985702 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.985705 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.989505 5073 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.989518 5073 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.989536 5073 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.989541 5073 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989609 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989614 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989617 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989620 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989623 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989626 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989628 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:36.000384 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989631 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989635 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989637 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989640 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989642 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989645 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989648 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989650 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989653 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989656 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989658 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989666 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989669 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989672 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989674 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989676 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989679 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989681 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989684 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:36.000869 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989686 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989689 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989691 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989694 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989696 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989698 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989701 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989703 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989706 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989708 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989710 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989713 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989715 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989719 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989723 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989726 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989729 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989732 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989734 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:36.001327 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989737 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989740 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989743 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989745 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989748 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989751 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989761 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989765 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989768 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989770 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989773 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989775 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989777 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989796 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989801 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989803 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989806 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989808 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989811 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:36.001801 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989813 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989816 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989818 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989821 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989823 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989826 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989828 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989830 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989833 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989836 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989838 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989841 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989843 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989846 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989849 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989852 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989854 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989857 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989860 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989862 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:36.002365 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989871 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989874 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989876 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989878 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989881 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989883 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989886 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989888 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989891 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989893 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989895 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989898 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989900 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989902 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989905 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989907 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989910 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989912 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989914 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989917 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989919 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:36.003017 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.989922 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.989949 5073 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990132 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990136 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990139 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990142 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990145 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990147 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990149 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990152 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990155 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990157 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990165 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990168 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990171 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990173 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:36.003688 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990175 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990178 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990180 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990183 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990185 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990187 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990190 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990193 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990195 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990207 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990210 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990212 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990215 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990217 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990220 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990222 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990225 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990227 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990231 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:36.004097 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990235 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990238 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990241 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990244 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990247 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990249 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990252 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990254 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990257 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990259 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990267 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990270 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990279 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990282 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990284 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990287 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990290 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990292 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990295 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:36.004637 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990297 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990300 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990302 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990304 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990307 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990309 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990312 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990314 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990317 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990319 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990321 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990324 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990326 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990329 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990331 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990334 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990336 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990339 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990341 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990344 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:36.005115 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990346 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990348 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990351 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990353 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990361 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990364 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990366 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990369 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990371 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990375 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990379 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990381 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990384 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990386 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990388 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990391 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990393 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990396 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990398 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:36.005593 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990400 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990403 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990405 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990407 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990410 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990412 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990415 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990418 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990420 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990423 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990425 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990428 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990430 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990433 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990435 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.990458 5073 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990505 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:36.006108 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990508 5073 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990517 5073 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990520 5073 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990522 5073 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990525 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990527 5073 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990530 5073 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990532 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990535 5073 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990537 5073 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990539 5073 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990542 5073 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990544 5073 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990546 5073 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990549 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990551 5073 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990554 5073 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990556 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990559 5073 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990561 5073 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:36.006532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990563 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990573 5073 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990580 5073 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990583 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990587 5073 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990589 5073 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990592 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990595 5073 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990597 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990600 5073 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990602 5073 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990605 5073 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990607 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990610 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990618 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990620 5073 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990623 5073 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990625 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990628 5073 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:36.007030 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990630 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990633 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990635 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990637 5073 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990640 5073 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990642 5073 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990645 5073 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990648 5073 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990650 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990653 5073 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990656 5073 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990658 5073 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990661 5073 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990663 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990666 5073 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990672 5073 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990675 5073 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990677 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990680 5073 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990682 5073 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:36.007532 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990685 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990687 5073 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990690 5073 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990692 5073 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990695 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990697 5073 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990700 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990702 5073 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990711 5073 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990714 5073 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990716 5073 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990721 5073 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990725 5073 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990728 5073 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990731 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990734 5073 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990737 5073 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990739 5073 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990742 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:36.008142 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990744 5073 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990746 5073 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990749 5073 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990751 5073 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990754 5073 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990756 5073 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990759 5073 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990761 5073 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990765 5073 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990768 5073 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990770 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990773 5073 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990775 5073 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990778 5073 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990798 5073 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990802 5073 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990806 5073 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990809 5073 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990811 5073 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990814 5073 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:36.008725 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990816 5073 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990818 5073 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990827 5073 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990830 5073 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990833 5073 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990836 5073 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:35.990840 5073 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.990893 5073 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.991862 5073 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.991914 5073 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.992114 5073 server.go:1014] "Starting client certificate rotation" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.992273 5073 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.992364 5073 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:52:54 +0000 UTC" deadline="2028-07-11 07:21:06.097117296 +0000 UTC" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.992429 5073 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14863h22m30.104690588s" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.992647 5073 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.995425 5073 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.996646 5073 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:58:36.009282 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.998353 5073 log.go:25] "Validated CRI v1 image API" Oct 30 23:58:36.009719 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:35.999152 5073 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:58:36.009719 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.001098 5073 fs.go:135] Filesystem UUIDs: map[75511820-59e9-4be9-bade-537393f2b359:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b5516c5b-58df-473c-954c-97caa7a37b96:/dev/nvme0n1p4] Oct 30 23:58:36.009835 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.001118 5073 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07d01ef2c2a8a18542a45e435249cfc41df999ff1b1d2496ec9ba336ce6ca08b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07d01ef2c2a8a18542a45e435249cfc41df999ff1b1d2496ec9ba336ce6ca08b/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14678713d30ae56b8d949a1122f2428e6956a5d14af1a67d12a4fb48d42860b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14678713d30ae56b8d949a1122f2428e6956a5d14af1a67d12a4fb48d42860b5/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c056bcec79eda73b369b99b7acc13686a621bf96b684c95d2f904da8a7db085/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c056bcec79eda73b369b99b7acc13686a621bf96b684c95d2f904da8a7db085/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fbe0f08468e86c474a57ebc457cab8c65e559ca14552f5e1ffb2cc1a86a26a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fbe0f08468e86c474a57ebc457cab8c65e559ca14552f5e1ffb2cc1a86a26a4/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63e174cc6e540a643b1b0660a64b97d1552042ec2d560daac796d571fda053a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63e174cc6e540a643b1b0660a64b97d1552042ec2d560daac796d571fda053a8/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/667d3f69c6bfa2fb317ec78ed34c0fc8687c98a92aeaae0c70abab59cf2244d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/667d3f69c6bfa2fb317ec78ed34c0fc8687c98a92aeaae0c70abab59cf2244d5/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7e017f89a6abe46ebe21ba510dd13018c20c01ecf83cd8717169b6e2ef3335b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7e017f89a6abe46ebe21ba510dd13018c20c01ecf83cd8717169b6e2ef3335b/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1adfb9f1b0640f1c50c36b6969ee19eeec796889773ba3a26c76f30798c53ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1adfb9f1b0640f1c50c36b6969ee19eeec796889773ba3a26c76f30798c53ae/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1a5c70626fea5bf132fdf22a1d05d73abd4621b6bd713a12836d3a12f7e3c71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1a5c70626fea5bf132fdf22a1d05d73abd4621b6bd713a12836d3a12f7e3c71/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/106281ea-1622-4966-bfd3-bdba5d770f5f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/106281ea-1622-4966-bfd3-bdba5d770f5f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~projected/kube-api-access-gmwjt:{mountpoint:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~projected/kube-api-access-gmwjt major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e72e0f4-7067-409f-bc3a-0dcff8e656fa/volumes/kubernetes.io~projected/kube-api-access-l7sgf:{mountpoint:/var/lib/kubelet/pods/1e72e0f4-7067-409f-bc3a-0dcff8e656fa/volumes/kubernetes.io~projected/kube-api-access-l7sgf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99/volumes/kubernetes.io~projected/kube-api-access-jgfc9:{mountpoint:/var/lib/kubelet/pods/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99/volumes/kubernetes.io~projected/kube-api-access-jgfc9 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volumes/kubernetes.io~projected/kube-api-access-wng4p:{mountpoint:/var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volumes/kubernetes.io~projected/kube-api-access-wng4p major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ed58e8f-b3ed-4c94-aebf-0676c621799d/volumes/kubernetes.io~projected/kube-api-access-26lcv:{mountpoint:/var/lib/kubelet/pods/3ed58e8f-b3ed-4c94-aebf-0676c621799d/volumes/kubernetes.io~projected/kube-api-access-26lcv major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c199e89-40d7-4a83-815c-f3533fcc31c4/volumes/kubernetes.io~projected/kube-api-access-x4m8w:{mountpoint:/var/lib/kubelet/pods/4c199e89-40d7-4a83-815c-f3533fcc31c4/volumes/kubernetes.io~projected/kube-api-access-x4m8w major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~projected/kube-api-access-jkrxj:{mountpoint:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~projected/kube-api-access-jkrxj major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fa50726-3415-4c7f-82ba-e6eb5115267a/volumes/kubernetes.io~projected/kube-api-access-vq5tn:{mountpoint:/var/lib/kubelet/pods/5fa50726-3415-4c7f-82ba-e6eb5115267a/volumes/kubernetes.io~projected/kube-api-access-vq5tn major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fa50726-3415-4c7f-82ba-e6eb5115267a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5fa50726-3415-4c7f-82ba-e6eb5115267a/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/686b1996-b6f2-44e9-b9bd-8acb156e833d/volumes/kubernetes.io~projected/kube-api-access-zq72r:{mountpoint:/var/lib/kubelet/pods/686b1996-b6f2-44e9-b9bd-8acb156e833d/volumes/kubernetes.io~projected/kube-api-access-zq72r major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6afa2442-496b-45d4-ae52-133e0d21a60f/volumes/kubernetes.io~projected/kube-api-access-wg79k:{mountpoint:/var/lib/kubelet/pods/6afa2442-496b-45d4-ae52-133e0d21a60f/volumes/kubernetes.io~projected/kube-api-access-wg79k major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6afa2442-496b-45d4-ae52-133e0d21a60f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6afa2442-496b-45d4-ae52-133e0d21a60f/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad/volumes/kubernetes.io~projected/kube-api-access-9kbt5:{mountpoint:/var/lib/kubelet/pods/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad/volumes/kubernetes.io~projected/kube-api-access-9kbt5 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~projected/kube-api-access-wjzqk:{mountpoint:/var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~projected/kube-api-access-wjzqk major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c479d0-cb02-4549-acb7-3a206eb2ecb7/volumes/kubernetes.io~projected/kube-api-access-9pgdj:{mountpoint:/var/lib/kubelet/pods/71c479d0-cb02-4549-acb7-3a206eb2ecb7/volumes/kubernetes.io~projected/kube-api-access-9pgdj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b/volumes/kubernetes.io~projected/kube-api-access-q9fxt:{mountpoint:/var/lib/kubelet/pods/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b/volumes/kubernetes.io~projected/kube-api-access-q9fxt major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e12f20-75a3-480a-81b1-f6bf080a6733/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/86e12f20-75a3-480a-81b1-f6bf080a6733/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a607b81-bbfa-4a4e-9828-2065d1e3bf49/volumes/kubernetes.io~projected/kube-api-access-lr8p8:{mountpoint:/var/lib/kubelet/pods/8a607b81-bbfa-4a4e-9828-2065d1e3bf49/volumes/kubernetes.io~projected/kube-api-access-lr8p8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c/volumes/kubernetes.io~projected/kube-api-access-vdl2z:{mountpoint:/var/lib/kubelet/pods/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c/volumes/kubernetes.io~projected/kube-api-access-vdl2z major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93866d6e-e48c-4e7f-823d-771497d4694b/volumes/kubernetes.io~projected/kube-api-access-x8kxc:{mountpoint:/var/lib/kubelet/pods/93866d6e-e48c-4e7f-823d-771497d4694b/volumes/kubernetes.io~projected/kube-api-access-x8kxc major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96ddf74c-7e70-4e9d-b5fd-77203f8d2179/volumes/kubernetes.io~projected/kube-api-access-pb57k:{mountpoint:/var/lib/kubelet/pods/96ddf74c-7e70-4e9d-b5fd-77203f8d2179/volumes/kubernetes.io~projected/kube-api-access-pb57k major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e/volumes/kubernetes.io~projected/kube-api-access-trvzk:{mountpoint:/var/lib/kubelet/pods/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e/volumes/kubernetes.io~projected/kube-api-access-trvzk major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1/volumes/kubernetes.io~projected/kube-api-access-kcnkw:{mountpoint:/var/lib/kubelet/pods/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1/volumes/kubernetes.io~projected/kube-api-access-kcnkw major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2528eea-3e1e-49eb-ad02-537965818be1/volumes/kubernetes.io~projected/kube-api-access-prpfp:{mountpoint:/var/lib/kubelet/pods/b2528eea-3e1e-49eb-ad02-537965818be1/volumes/kubernetes.io~projected/kube-api-access-prpfp major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2528eea-3e1e-49eb-ad02-537965818be1/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b2528eea-3e1e-49eb-ad02-537965818be1/volumes/kubernetes.io~secret/signing-key major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b31374f9-c1aa-4288-a5db-c88ec5b3f732/volumes/kubernetes.io~projected/kube-api-access-xvx55:{mountpoint:/var/lib/kubelet/pods/b31374f9-c1aa-4288-a5db-c88ec5b3f732/volumes/kubernetes.io~projected/kube-api-access-xvx55 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b470b4c4-501f-4d51-a04d-37f1f3d71c06/volumes/kubernetes.io~projected/kube-api-access-ztxcg:{mountpoint:/var/lib/kubelet/pods/b470b4c4-501f-4d51-a04d-37f1f3d71c06/volumes/kubernetes.io~projected/kube-api-access-ztxcg major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b613d66f-e947-427e-9694-3f0b609f35c4/volumes/kubernetes.io~projected/kube-api-access-jdchj:{mountpoint:/var/lib/kubelet/pods/b613d66f-e947-427e-9694-3f0b609f35c4/volumes/kubernetes.io~projected/kube-api-access-jdchj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/beff8893-2560-45f8-a96d-bce1e51dcf80/volumes/kubernetes.io~projected/kube-api-access-5jjjd:{mountpoint:/var/lib/kubelet/pods/beff8893-2560-45f8-a96d-bce1e51dcf80/volumes/kubernetes.io~projected/kube-api-access-5jjjd major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/beff8893-2560-45f8-a96d-bce1e51dcf80/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/beff8893-2560-45f8-a96d-bce1e51dcf80/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ffac65-ad93-4617-afbd-9d2fe6153391/volumes/kubernetes.io~projected/kube-api-access-wb5l2:{mountpoint:/var/lib/kubelet/pods/d0ffac65-ad93-4617-afbd-9d2fe6153391/volumes/kubernetes.io~projected/kube-api-access-wb5l2 major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d91a42d7-9685-4f8f-b620-5e93d58b1a76/volumes/kubernetes.io~projected/kube-api-access-dnw7z:{mountpoint:/var/lib/kubelet/pods/d91a42d7-9685-4f8f-b620-5e93d58b1a76/volumes/kubernetes.io~projected/kube-api-access-dnw7z major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1a27ea3-18d4-4aff-a92b-a4290ec72714/volumes/kubernetes.io~projected/kube-api-access-6jkh6:{mountpoint:/var/lib/kubelet/pods/e1a27ea3-18d4-4aff-a92b-a4290ec72714/volumes/kubernetes.io~projected/kube-api-access-6jkh6 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~projected/kube-api-access-67z2d:{mountpoint:/var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~projected/kube-api-access-67z2d major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f270b734-9691-480f-8d55-1448c307a69b/volumes/kubernetes.io~projected/kube-api-access-8fwwx:{mountpoint:/var/lib/kubelet/pods/f270b734-9691-480f-8d55-1448c307a69b/volumes/kubernetes.io~projected/kube-api-access-8fwwx major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f270b734-9691-480f-8d55-1448c307a69b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f270b734-9691-480f-8d55-1448c307a69b/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbea15b0-87b2-480b-9bb7-496f71990632/volumes/kubernetes.io~projected/kube-api-access-j6sbf:{mountpoint:/var/lib/kubelet/pods/fbea15b0-87b2-480b-9bb7-496f71990632/volumes/kubernetes.io~projected/kube-api-access-j6sbf major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fd99ff7df0bda84072a42505b5948708592dd63983c29cb3418202a3019a852c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/035eb706db6749183f6ed2d6765c162ec3f680c0341cfb13cf6d40f8660e8854/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1e2aa1021951507d94e2e79372472183878b6acf60833d25332d2afa135dde3e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/d6f310c3fcf3ebde9264a397518a2ae75366290bab23be163a2119285ffb9ef2/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/52e43f2755a2be53f2f1af437a6d02f0be92da0245e96f11cc9fa201fa68f522/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/232d13bbbfc85fba09e7eefcd0189e20d7816a293f341f056149f52ecc24a3ff/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0b9d26f6aefd9c0f7c21a817d200016aa3dcf130f95f9f24e196d70174b38586/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/893e20b4b84b8b20b352f409884823ea41299e75cc971e10babfcbd7267872c6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4a4dbc0d0e93855e13347bdfa1f863339c624ab2b79c63d1246b95ddeb3c4e98/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d069b4bdf837089671987ef91e635dcbe6a843437f4d6f2933d00da71acfcc0d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/9405068b935c67bbf0313b01bb9332a589d643528c1abb6719e4c369a61bb8ca/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/6c75ea5d4bc9e311758839d172436597e28428eff6a74cf756545b935b55c0eb/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/bbb8afd170268978288b43c848ef4216cdbf70e6172b8baf8003a2faa299ab69/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/76b220852c7e3917986c33254f732b34e7c332fc21e9b18750d7459a2fff593a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/da6069503c0381e595930295d3b9e39c8f8e91348fbb25517746e0dc684264f2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/32c91652c9b745859afb4d119aea800c92b8f42eb62002a6788e7a3c2f85e3a7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/83acfc787994abf6bba8d8658a8b28aff63a5fb3eaddf0cadb5d6828f3d604af/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7c350016cc601e4ed76108609f4f1ecad3de95de69beb56f0bdd2e0d78a2a532/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0f3f4ae0722cb7bc4aee59df4da1d577b41710f34659025e4241c710c6acff84/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9174632f782271fa8310fe6c0c0a1f8295896945eecb6146a7d7b339b9aaf772/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/691a15f1d8e636745d75b0011f31f2b02eaaa1bb285539f1defdaf4ff23de46c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b07522a9061fee847c58452aa9b4a307363416fe45b89b14850896705d33e5be/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/89b4602f17cf7e792e2b483e80993b607de3cdbeb2a4b6d1616e7deb9e71a47c/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/acdd018bcfc47ac674da261244e9f89ed27487e2d29f8c8e2e42cf2f429a6426/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/6d2273670822b05b97ddf74c3f0ad693d48eb23699432d447b559bbf89cf419e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/93b23e2819f2f25785a97e66e521ba415a19633eedf690f899f0d05b4cfc3c98/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c3c26318febe94031c37ee0786aa312ec6af1769a1744d53135c586c2ec74ff1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/cdfe661890227484401c93ee63e2d00fc83ecb29664101da555e64484d35a98d/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/a08aef4c454c925434090f8d67bbb37e61b671f56a775363513c0f3f2c44db7d/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/c4e7638d4d57463855a6a7fa8e5efdf10820545799dbd65b8e4464a5d0bf25b0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/248050b5e1daa0846869e9c64cb9e24cfb0fc609734f8530ddbe347c783f1d6a/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/1ae3d87db61bd8f86a433d831ad3d76ab3da4b82384ac55074dffe2c25cbb4a1/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/744e5cc39a81d4863ef2bb97c19243184b6f68b757556770155e4323737b8f15/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/c1c104f5b787079442a17a4f039af5464dc859408ab7dd30de2117cdb9ceb4b3/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/0149f62bfaab28755a12c954d7d3a16484fec9a1aff1ed966572ab04a872aaf9/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/f9700c63c18eb18ed22733fc8747cab848d4ea5339f80927f3bc4c6676c8b50d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/630e9214ebbdb13a1a654fd9922a86ae8d7dcd195982ebb53951837a94fe4319/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/4aa102d00439dafff8e1802f76322d6df4bb25a26529384be0a227495b5817c9/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/60e79b5ada82fa04c70e43a32f3762edf658ab7a3ffccaa20a5f3f67e607a58e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c945ad68b608375d77da5356c6bd4cd4933049333300d13d13ef870d934ee7a6/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/3711062561f5b62194a2088c7b890832b75f56f5f188a62dc5449d1c3156a0ac/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/1bdecf1d8433567c9e6ddb520a27117514f9cc3ea3fc92dfee5e9184acc8fa36/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/d53f177b3b3f69f20a71851deba10acf6ac077190f54159d7253e1e5172209fa/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/b3fae59112bbaab4fbf552a3273774d34c911e79d7f63a573f92f446fe962c00/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/7f4a48d8b9363cf030c23a285d891f51679ff793ca862251b4d73bb06d5e7b4b/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/3949e87359f484f7552d4e74f37e15bc5cd03a9c6348b0612ec59aafe2144a8b/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/5516b45639a4a2c374bc1bca86d062dfe8110cd7056796612abe15ec35fd5ea2/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/7ee094d04d37530f8403abab445c6aa5930b6acdfb415603a0cb5be7e653c20f/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/17f4c795ecfb2f64d1f746a356cbcc3fa56b889a2f14bdf2e892240613daab52/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/55755adc8f95ed855726e773f2490e1c709e1aeacd23b0a9c5e09a85e3b748e7/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/213f35fb16a855a1bdb92a1eed9e6008ab83ca853af1e780bd9e8d33116dbc21/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/47329211ff8512414fae3b0cd219b62ec7bea0e6a31a6b7630fd3d35d89d19a5/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/371c2907eb65584288ef61e144763ff00d6d43cc5ab42733f101bd9189ca71c0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6df3b51756b80a97fb7a39a59e939525996dc9436b5976ddef3f12391ada0d29/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f042d9e8b77d9cf5abf09f40102f8ce0e197d559cad345387122711456d4d4ea/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2f984655cfc8ecb805f19f2793db559d1beb72cf905bb0cadb0347bf90d452c5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1838449966cdb9197b847dd42acc89b22455242f99dc9e01889a66f6f45bc2f9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6ad8705ecdd309833ae459e37cca0db644ad747ff65efcd618e4486f479df719/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0fa3e269d8714af862829c64e876aca32fc9c10812725abd55f7725a5770c1e3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/822953ba387da8ebc7ad6952d8d5e21f475da5a614448b1adcf65a281c19a5f9/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/9d4bde9b362544eefc2f91babe53dd287a72be7983ab97dd3dd2d3a193fe9370/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e1cfed6a4ad27a6ac99e909682d6bd65e56cb6b9a8a5b7556c096007a67bef31/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.009232 5073 manager.go:217] Machine: {Timestamp:2026-10-30 23:58:36.008763089 +0000 UTC m=+0.069537775 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126188 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23a0ed0ab7f9719b607349b13b2348 SystemUUID:ec23a0ed-0ab7-f971-9b60-7349b13b2348 BootID:c807b218-bcf5-41cb-b775-541c261804bf Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d307e7e16fa025c86edc99101dee8e37fdf011ca91911a97a08015a085164709/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~projected/kube-api-access-jkrxj DeviceMajor:0 DeviceMinor:229 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/667d3f69c6bfa2fb317ec78ed34c0fc8687c98a92aeaae0c70abab59cf2244d5/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/1e72e0f4-7067-409f-bc3a-0dcff8e656fa/volumes/kubernetes.io~projected/kube-api-access-l7sgf DeviceMajor:0 DeviceMinor:58 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/fbea15b0-87b2-480b-9bb7-496f71990632/volumes/kubernetes.io~projected/kube-api-access-j6sbf DeviceMajor:0 DeviceMinor:65 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3387f86ceaadaa35b3404f26ebeea0ac48cc0659727f2a8e088a31bb488ebcc/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07d01ef2c2a8a18542a45e435249cfc41df999ff1b1d2496ec9ba336ce6ca08b/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ff405015d2cc9e69e9c425c8bb55a089409a3e30ab647f7381b6d4067f4195f/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed1459500479c70a13cc5c0b12e6221cdccdaa0a75e9ad0119ba1ee362b42356/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99/volumes/kubernetes.io~projected/kube-api-access-jgfc9 DeviceMajor:0 DeviceMinor:207 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/f270b734-9691-480f-8d55-1448c307a69b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/5fa50726-3415-4c7f-82ba-e6eb5115267a/volumes/kubernetes.io~projected/kube-api-access-vq5tn DeviceMajor:0 DeviceMinor:224 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~projected/kube-api-access-wjzqk DeviceMajor:0 DeviceMinor:213 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63e174cc6e540a643b1b0660a64b97d1552042ec2d560daac796d571fda053a8/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~projected/kube-api-access-gmwjt DeviceMajor:0 DeviceMinor:209 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b664a05c475329144520bdf378cdb64e2676794e8aab1d417d010bc41eff5a79/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad/volumes/kubernetes.io~projected/kube-api-access-9kbt5 DeviceMajor:0 DeviceMinor:206 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1a27ea3-18d4-4aff-a92b-a4290ec72714/volumes/kubernetes.io~projected/kube-api-access-6jkh6 DeviceMajor:0 DeviceMinor:225 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/686b1996-b6f2-44e9-b9bd-8acb156e833d/volumes/kubernetes.io~projected/kube-api-access-zq72r DeviceMajor:0 DeviceMinor:316 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volumes/kubernetes.io~projected/kube-api-access-wng4p DeviceMajor:0 DeviceMinor:57 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/4c199e89-40d7-4a83-815c-f3533fcc31c4/volumes/kubernetes.io~projected/kube-api-access-x4m8w DeviceMajor:0 DeviceMinor:66 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~projected/kube-api-access-67z2d DeviceMajor:0 DeviceMinor:67 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/381ecd5c275146d11be75f3401e5dba09c5b9083613132bae7f8b29673fcbae2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/749f82eb31a29f02e9739e23edf03a87a4a8e2b0caa5dfbc444abb91722e0b90/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ffac65-ad93-4617-afbd-9d2fe6153391/volumes/kubernetes.io~projected/kube-api-access-wb5l2 DeviceMajor:0 DeviceMinor:321 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1a5c70626fea5bf132fdf22a1d05d73abd4621b6bd713a12836d3a12f7e3c71/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c056bcec79eda73b369b99b7acc13686a621bf96b684c95d2f904da8a7db085/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96ddf74c-7e70-4e9d-b5fd-77203f8d2179/volumes/kubernetes.io~projected/kube-api-access-pb57k DeviceMajor:0 DeviceMinor:60 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d91a42d7-9685-4f8f-b620-5e93d58b1a76/volumes/kubernetes.io~projected/kube-api-access-dnw7z DeviceMajor:0 DeviceMinor:347 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71c479d0-cb02-4549-acb7-3a206eb2ecb7/volumes/kubernetes.io~projected/kube-api-access-9pgdj DeviceMajor:0 DeviceMinor:61 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/b2528eea-3e1e-49eb-ad02-537965818be1/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:342 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/93866d6e-e48c-4e7f-823d-771497d4694b/volumes/kubernetes.io~projected/kube-api-access-x8kxc DeviceMajor:0 DeviceMinor:212 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/3ed58e8f-b3ed-4c94-aebf-0676c621799d/volumes/kubernetes.io~projected/kube-api-access-26lcv DeviceMajor:0 DeviceMinor:214 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c/volumes/kubernetes.io~projected/kube-api-access-vdl2z DeviceMajor:0 DeviceMinor:227 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/106281ea-1622-4966-bfd3-bdba5d770f5f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/6df4d978-5bb3-48e6-a69f-519f0d85e465/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:219 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc40da3b6863462f96567f73bc1c98fa99f4d47d61ecc09823f66503847b7eb3/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b31374f9-c1aa-4288-a5db-c88ec5b3f732/volumes/kubernetes.io~projected/kube-api-access-xvx55 DeviceMajor:0 DeviceMinor:295 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/f270b734-9691-480f-8d55-1448c307a69b/volumes/kubernetes.io~projected/kube-api-access-8fwwx DeviceMajor:0 DeviceMinor:226 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b470b4c4-501f-4d51-a04d-37f1f3d71c06/volumes/kubernetes.io~projected/kube-api-access-ztxcg DeviceMajor:0 DeviceMinor:215 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/5fa50726-3415-4c7f-82ba-e6eb5115267a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6afa2442-496b-45d4-ae52-133e0d21a60f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7e017f89a6abe46ebe21ba510dd13018c20c01ecf83cd8717169b6e2ef3335b/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95296eeee05e98666a247cc9e5124a1dc73a142fba171e7ecccbf879f94ceb5d/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e12f20-75a3-480a-81b1-f6bf080a6733/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/b613d66f-e947-427e-9694-3f0b609f35c4/volumes/kubernetes.io~projected/kube-api-access-jdchj DeviceMajor:0 DeviceMinor:59 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1adfb9f1b0640f1c50c36b6969ee19eeec796889773ba3a26c76f30798c53ae/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1/volumes/kubernetes.io~projected/kube-api-access-kcnkw DeviceMajor:0 DeviceMinor:343 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f8f35727c2e9b69124d24712de69589a6c7235c4ae68c5a6ca84364f14ef912/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2528eea-3e1e-49eb-ad02-537965818be1/volumes/kubernetes.io~projected/kube-api-access-prpfp DeviceMajor:0 DeviceMinor:344 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3db5e05a-532e-4937-a241-3a995c5c43ae/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e/volumes/kubernetes.io~projected/kube-api-access-trvzk DeviceMajor:0 DeviceMinor:211 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14678713d30ae56b8d949a1122f2428e6956a5d14af1a67d12a4fb48d42860b5/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a607b81-bbfa-4a4e-9828-2065d1e3bf49/volumes/kubernetes.io~projected/kube-api-access-lr8p8 DeviceMajor:0 DeviceMinor:62 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19bce3388341f6f1555649e6be0506df6b7aa63083fe84eff2a80f0a5ec81c6f/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9923127abbb795ed24013c9831ee2a7a3966a3b6797a510817d3645bb078620/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/beff8893-2560-45f8-a96d-bce1e51dcf80/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/beff8893-2560-45f8-a96d-bce1e51dcf80/volumes/kubernetes.io~projected/kube-api-access-5jjjd DeviceMajor:0 DeviceMinor:208 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fbe0f08468e86c474a57ebc457cab8c65e559ca14552f5e1ffb2cc1a86a26a4/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/e3036952-ce27-471a-9fa5-933ea133e22e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/159ede2993365e0688b95b0ddaff2930b770387559473a56f78019e3c13693e8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21539906dda915a114d5edc62d8efc7e3aeb9c9bdc65651d1001b9f87944044e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b/volumes/kubernetes.io~projected/kube-api-access-q9fxt DeviceMajor:0 DeviceMinor:205 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/6afa2442-496b-45d4-ae52-133e0d21a60f/volumes/kubernetes.io~projected/kube-api-access-wg79k DeviceMajor:0 DeviceMinor:228 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31615fd0add2106a165907f59a63d0bd747dd0f27cdb7a6e9a04612b42d998c3/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07d01ef2c2a8a18 MacAddress:6a:a1:f0:25:6e:c8 Speed:10000 Mtu:8901} {Name:14678713d30ae56 MacAddress:c2:45:74:41:6d:31 Speed:10000 Mtu:8901} {Name:2c056bcec79eda7 MacAddress:8a:4c:ee:86:32:66 Speed:10000 Mtu:8901} {Name:2fbe0f08468e86c MacAddress:d6:fd:e6:6f:96:96 Speed:10000 Mtu:8901} {Name:31615fd0add2106 MacAddress:da:b2:1b:99:e3:c4 Speed:10000 Mtu:8901} {Name:3ff405015d2cc9e MacAddress:be:7f:32:98:6d:d3 Speed:10000 Mtu:8901} {Name:63e174cc6e540a6 MacAddress:3e:6c:55:ef:20:68 Speed:10000 Mtu:8901} {Name:667d3f69c6bfa2f MacAddress:fe:74:92:a4:24:bd Speed:10000 Mtu:8901} {Name:95296eeee05e986 MacAddress:ce:09:57:0d:7d:79 Speed:10000 Mtu:8901} {Name:a7e017f89a6abe4 MacAddress:b2:7f:bc:c6:a5:5a Speed:10000 Mtu:8901} {Name:b1adfb9f1b0640f MacAddress:06:b1:a3:fe:f0:d8 Speed:10000 Mtu:8901} {Name:b664a05c4753291 MacAddress:f2:9a:bd:4d:3a:81 Speed:10000 Mtu:8901} {Name:b9923127abbb795 MacAddress:8e:09:fd:11:82:fa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ce:e2:a4:5d Speed:0 Mtu:9001} {Name:br-int MacAddress:02:83:3d:fa:cf:43 Speed:0 Mtu:8901} {Name:c1a5c70626fea5b MacAddress:1a:82:ec:66:b3:1b Speed:10000 Mtu:8901} {Name:c3387f86ceaadaa MacAddress:6e:a1:4f:a5:3e:4f Speed:10000 Mtu:8901} {Name:dc40da3b6863462 MacAddress:9a:f6:bc:5a:56:46 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ce:e2:a4:5d Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:39:3a:6c:87:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.009583 5073 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.009621 5073 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.009905 5073 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.009923 5073 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-246.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010048 5073 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010055 5073 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010064 5073 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:58:36.010076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010077 5073 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010095 5073 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010251 5073 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010342 5073 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010373 5073 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010382 5073 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010391 5073 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010400 5073 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:58:36.010553 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.010408 5073 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:58:36.012433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.011578 5073 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:58:36.012433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.011596 5073 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:58:36.012599 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.012577 5073 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:58:36.015623 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.015602 5073 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:58:36.015623 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.015617 5073 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:58:36.015756 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.015713 5073 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:58:36.015924 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.015908 5073 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:58:36.015958 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.015936 5073 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:58:36.016029 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016020 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016034 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016040 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016046 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016051 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016057 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016062 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016067 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:58:36.016070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016073 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:58:36.016385 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016079 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:58:36.016385 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016088 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:58:36.016385 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016096 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:58:36.016385 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016122 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:58:36.016385 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.016128 5073 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:58:36.019561 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.019548 5073 server.go:1263] "Started kubelet" Oct 30 23:58:36.019656 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.019607 5073 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:58:36.019715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.019665 5073 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:58:36.019764 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.019736 5073 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:58:36.020087 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.020071 5073 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:58:36.020113 ip-10-0-140-246 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:58:36.022207 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.022164 5073 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:58:36.022960 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.022939 5073 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:58:36.023071 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.023049 5073 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:58:36.023600 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.023190 5073 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:52:56 +0000 UTC" deadline="2028-05-07 23:14:03.340668681 +0000 UTC" Oct 30 23:58:36.023600 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.023216 5073 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13319h15m27.31745635s" Oct 30 23:58:36.023600 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.023269 5073 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:58:36.023600 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.023279 5073 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:58:36.023600 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.023357 5073 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:58:36.023600 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.023338 5073 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-246.us-east-2.compute.internal\" not found" Oct 30 23:58:36.025952 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.025934 5073 factory.go:55] Registering systemd factory Oct 30 23:58:36.025952 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.025955 5073 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:58:36.028736 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.028204 5073 factory.go:153] Registering CRI-O factory Oct 30 23:58:36.028736 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.028223 5073 factory.go:223] Registration of the crio container factory successfully Oct 30 23:58:36.028736 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.028274 5073 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:58:36.028736 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.028320 5073 factory.go:103] Registering Raw factory Oct 30 23:58:36.028736 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.028345 5073 manager.go:1199] Started watching for new ooms in manager Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.030049 5073 manager.go:319] Starting recovery of all containers Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.033924 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fa50726-3415-4c7f-82ba-e6eb5115267a" volumeName="kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.033955 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a607b81-bbfa-4a4e-9828-2065d1e3bf49" volumeName="kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-daemon-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.033970 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2528eea-3e1e-49eb-ad02-537965818be1" volumeName="kubernetes.io/secret/b2528eea-3e1e-49eb-ad02-537965818be1-signing-key" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.033984 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1a27ea3-18d4-4aff-a92b-a4290ec72714" volumeName="kubernetes.io/empty-dir/e1a27ea3-18d4-4aff-a92b-a4290ec72714-tmp-dir" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.033996 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3036952-ce27-471a-9fa5-933ea133e22e" volumeName="kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-etc-tuned" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034081 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034093 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dbbbfcf-c966-49b9-8cf3-91b77469bd2b" volumeName="kubernetes.io/configmap/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-telemetry-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034105 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3036952-ce27-471a-9fa5-933ea133e22e" volumeName="kubernetes.io/projected/e3036952-ce27-471a-9fa5-933ea133e22e-kube-api-access-67z2d" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034129 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbea15b0-87b2-480b-9bb7-496f71990632" volumeName="kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034142 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034154 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034165 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c479d0-cb02-4549-acb7-3a206eb2ecb7" volumeName="kubernetes.io/projected/71c479d0-cb02-4549-acb7-3a206eb2ecb7-kube-api-access-9pgdj" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034178 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93866d6e-e48c-4e7f-823d-771497d4694b" volumeName="kubernetes.io/projected/93866d6e-e48c-4e7f-823d-771497d4694b-kube-api-access-x8kxc" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034216 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d91a42d7-9685-4f8f-b620-5e93d58b1a76" volumeName="kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034232 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d91a42d7-9685-4f8f-b620-5e93d58b1a76" volumeName="kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034257 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034270 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b470b4c4-501f-4d51-a04d-37f1f3d71c06" volumeName="kubernetes.io/projected/b470b4c4-501f-4d51-a04d-37f1f3d71c06-kube-api-access-ztxcg" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034282 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d91a42d7-9685-4f8f-b620-5e93d58b1a76" volumeName="kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034293 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbea15b0-87b2-480b-9bb7-496f71990632" volumeName="kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-binary-copy" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034335 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ed58e8f-b3ed-4c94-aebf-0676c621799d" volumeName="kubernetes.io/projected/3ed58e8f-b3ed-4c94-aebf-0676c621799d-kube-api-access-26lcv" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034348 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034361 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fa50726-3415-4c7f-82ba-e6eb5115267a" volumeName="kubernetes.io/secret/5fa50726-3415-4c7f-82ba-e6eb5115267a-serving-cert" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034373 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad" volumeName="kubernetes.io/configmap/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034385 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a607b81-bbfa-4a4e-9828-2065d1e3bf49" volumeName="kubernetes.io/projected/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-kube-api-access-lr8p8" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034397 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeeea8e9-d19d-4518-b9cb-285b96d9a5d1" volumeName="kubernetes.io/projected/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-api-access-kcnkw" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034409 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b613d66f-e947-427e-9694-3f0b609f35c4" volumeName="kubernetes.io/empty-dir/b613d66f-e947-427e-9694-3f0b609f35c4-tmp-dir" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034421 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beff8893-2560-45f8-a96d-bce1e51dcf80" volumeName="kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-snapshots" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034432 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df4d978-5bb3-48e6-a69f-519f0d85e465" volumeName="kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-default-certificate" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034450 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dbbbfcf-c966-49b9-8cf3-91b77469bd2b" volumeName="kubernetes.io/projected/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-kube-api-access-q9fxt" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034462 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e12f20-75a3-480a-81b1-f6bf080a6733" volumeName="kubernetes.io/configmap/86e12f20-75a3-480a-81b1-f6bf080a6733-konnectivity-ca" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034474 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96ddf74c-7e70-4e9d-b5fd-77203f8d2179" volumeName="kubernetes.io/projected/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-kube-api-access-pb57k" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034487 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2587ab9-e842-47c6-87b5-fb0b1ab8b76e" volumeName="kubernetes.io/projected/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-kube-api-access-trvzk" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034498 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeeea8e9-d19d-4518-b9cb-285b96d9a5d1" volumeName="kubernetes.io/empty-dir/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-data-volume" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034509 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beff8893-2560-45f8-a96d-bce1e51dcf80" volumeName="kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-tmp" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034523 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beff8893-2560-45f8-a96d-bce1e51dcf80" volumeName="kubernetes.io/projected/beff8893-2560-45f8-a96d-bce1e51dcf80-kube-api-access-5jjjd" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034534 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2587ab9-e842-47c6-87b5-fb0b1ab8b76e" volumeName="kubernetes.io/secret/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-serving-cert" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034547 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fa50726-3415-4c7f-82ba-e6eb5115267a" volumeName="kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-trusted-ca" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034558 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034572 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034583 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034596 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034609 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2f8fec0-412d-40ee-9d94-db3fbe04a651" volumeName="kubernetes.io/configmap/b2f8fec0-412d-40ee-9d94-db3fbe04a651-nginx-conf" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034623 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ffac65-ad93-4617-afbd-9d2fe6153391" volumeName="kubernetes.io/projected/d0ffac65-ad93-4617-afbd-9d2fe6153391-kube-api-access-wb5l2" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034634 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db5e05a-532e-4937-a241-3a995c5c43ae" volumeName="kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-env-overrides" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034646 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b613d66f-e947-427e-9694-3f0b609f35c4" volumeName="kubernetes.io/projected/b613d66f-e947-427e-9694-3f0b609f35c4-kube-api-access-jdchj" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034657 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbea15b0-87b2-480b-9bb7-496f71990632" volumeName="kubernetes.io/projected/fbea15b0-87b2-480b-9bb7-496f71990632-kube-api-access-j6sbf" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034671 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034682 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3da2a0da-b9b7-41f0-8e8d-6e07faa06d99" volumeName="kubernetes.io/projected/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99-kube-api-access-jgfc9" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034694 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db5e05a-532e-4937-a241-3a995c5c43ae" volumeName="kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034706 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c199e89-40d7-4a83-815c-f3533fcc31c4" volumeName="kubernetes.io/projected/4c199e89-40d7-4a83-815c-f3533fcc31c4-kube-api-access-x4m8w" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034718 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e12f20-75a3-480a-81b1-f6bf080a6733" volumeName="kubernetes.io/secret/86e12f20-75a3-480a-81b1-f6bf080a6733-agent-certs" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034731 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d91a42d7-9685-4f8f-b620-5e93d58b1a76" volumeName="kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034743 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fa50726-3415-4c7f-82ba-e6eb5115267a" volumeName="kubernetes.io/projected/5fa50726-3415-4c7f-82ba-e6eb5115267a-kube-api-access-vq5tn" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034755 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df4d978-5bb3-48e6-a69f-519f0d85e465" volumeName="kubernetes.io/projected/6df4d978-5bb3-48e6-a69f-519f0d85e465-kube-api-access-wjzqk" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034767 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df4d978-5bb3-48e6-a69f-519f0d85e465" volumeName="kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-stats-auth" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034779 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1a27ea3-18d4-4aff-a92b-a4290ec72714" volumeName="kubernetes.io/configmap/e1a27ea3-18d4-4aff-a92b-a4290ec72714-config-volume" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034817 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1a27ea3-18d4-4aff-a92b-a4290ec72714" volumeName="kubernetes.io/projected/e1a27ea3-18d4-4aff-a92b-a4290ec72714-kube-api-access-6jkh6" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034830 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e72e0f4-7067-409f-bc3a-0dcff8e656fa" volumeName="kubernetes.io/projected/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kube-api-access-l7sgf" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034843 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034863 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="106281ea-1622-4966-bfd3-bdba5d770f5f" volumeName="kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034875 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db5e05a-532e-4937-a241-3a995c5c43ae" volumeName="kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034887 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6afa2442-496b-45d4-ae52-133e0d21a60f" volumeName="kubernetes.io/configmap/6afa2442-496b-45d4-ae52-133e0d21a60f-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034900 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6afa2442-496b-45d4-ae52-133e0d21a60f" volumeName="kubernetes.io/secret/6afa2442-496b-45d4-ae52-133e0d21a60f-serving-cert" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034912 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2587ab9-e842-47c6-87b5-fb0b1ab8b76e" volumeName="kubernetes.io/configmap/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-config" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034926 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2528eea-3e1e-49eb-ad02-537965818be1" volumeName="kubernetes.io/configmap/b2528eea-3e1e-49eb-ad02-537965818be1-signing-cabundle" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034938 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beff8893-2560-45f8-a96d-bce1e51dcf80" volumeName="kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034951 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c479d0-cb02-4549-acb7-3a206eb2ecb7" volumeName="kubernetes.io/configmap/71c479d0-cb02-4549-acb7-3a206eb2ecb7-serviceca" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034963 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b31374f9-c1aa-4288-a5db-c88ec5b3f732" volumeName="kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034974 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3036952-ce27-471a-9fa5-933ea133e22e" volumeName="kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-tmp" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.034986 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f270b734-9691-480f-8d55-1448c307a69b" volumeName="kubernetes.io/projected/f270b734-9691-480f-8d55-1448c307a69b-kube-api-access-8fwwx" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035000 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad" volumeName="kubernetes.io/projected/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-kube-api-access-9kbt5" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035011 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beff8893-2560-45f8-a96d-bce1e51dcf80" volumeName="kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-service-ca-bundle" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035030 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="beff8893-2560-45f8-a96d-bce1e51dcf80" volumeName="kubernetes.io/secret/beff8893-2560-45f8-a96d-bce1e51dcf80-serving-cert" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035063 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f270b734-9691-480f-8d55-1448c307a69b" volumeName="kubernetes.io/secret/f270b734-9691-480f-8d55-1448c307a69b-cert" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035076 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035090 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db5e05a-532e-4937-a241-3a995c5c43ae" volumeName="kubernetes.io/projected/3db5e05a-532e-4937-a241-3a995c5c43ae-kube-api-access-wng4p" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035105 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db5e05a-532e-4937-a241-3a995c5c43ae" volumeName="kubernetes.io/secret/3db5e05a-532e-4937-a241-3a995c5c43ae-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035116 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="686b1996-b6f2-44e9-b9bd-8acb156e833d" volumeName="kubernetes.io/projected/686b1996-b6f2-44e9-b9bd-8acb156e833d-kube-api-access-zq72r" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035194 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6afa2442-496b-45d4-ae52-133e0d21a60f" volumeName="kubernetes.io/projected/6afa2442-496b-45d4-ae52-133e0d21a60f-kube-api-access-wg79k" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035213 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a607b81-bbfa-4a4e-9828-2065d1e3bf49" volumeName="kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cni-binary-copy" seLinuxMountContext="" Oct 30 23:58:36.035158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035227 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c" volumeName="kubernetes.io/projected/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c-kube-api-access-vdl2z" seLinuxMountContext="" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035240 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2528eea-3e1e-49eb-ad02-537965818be1" volumeName="kubernetes.io/projected/b2528eea-3e1e-49eb-ad02-537965818be1-kube-api-access-prpfp" seLinuxMountContext="" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035252 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10737b2e-c98b-433d-a846-c087bfc5410f" volumeName="kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted" seLinuxMountContext="" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035263 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c199e89-40d7-4a83-815c-f3533fcc31c4" volumeName="kubernetes.io/configmap/4c199e89-40d7-4a83-815c-f3533fcc31c4-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035275 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="500001f6-9b33-4347-b5f5-cfcf79e6ed09" volumeName="kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets" seLinuxMountContext="" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035288 5073 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbea15b0-87b2-480b-9bb7-496f71990632" volumeName="kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035299 5073 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035815 5073 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:36.038739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.035306 5073 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:58:36.046560 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.046530 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 30 23:58:36.047827 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.047436 5073 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:36.047827 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.047472 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 30 23:58:36.049214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.049193 5073 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:58:36.049333 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.049314 5073 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:36.050626 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.050594 5073 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:58:36.052103 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.052081 5073 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:58:36.052103 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.052106 5073 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:58:36.052254 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.052125 5073 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:58:36.052254 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.052159 5073 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:58:36.057298 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.057224 5073 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:36.058478 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.058460 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:36.059841 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.059763 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:36.074067 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.074008 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 30 23:58:36.074416 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.074381 5073 generic.go:356] "Generic (PLEG): container finished" podID="3db5e05a-532e-4937-a241-3a995c5c43ae" containerID="8c1c07030f27bd62d0112873e0e92da03390264eccfc3fda7474754c53eaa70f" exitCode=1 Oct 30 23:58:36.087343 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.087292 5073 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="932504f980adca498715687d429473dab65cec5074590dc186cce3f64d459bde" exitCode=0 Oct 30 23:58:36.087343 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.087318 5073 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="7868d83c8ec610ef29397061bce7f09a065050dafb51a6b1de906b9fd19b07d3" exitCode=0 Oct 30 23:58:36.087343 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.087325 5073 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="8e1b6f218807fd9da4418d1dd4f997f897d99b433ef27e06c015bb696efd5a79" exitCode=0 Oct 30 23:58:36.087343 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.087333 5073 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="f9c97d076f5eb95851e044e3bb06f40de010c44d014a14843ad644af652c754e" exitCode=0 Oct 30 23:58:36.087343 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.087344 5073 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="f253f398afef0f29cd962016af2a4c0b9f86914a1f7fb43457dcb5cd213c498c" exitCode=0 Oct 30 23:58:36.087582 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.087351 5073 generic.go:356] "Generic (PLEG): container finished" podID="fbea15b0-87b2-480b-9bb7-496f71990632" containerID="716947db0ad59cd9daa3581e2e28533f8f0820d096f9bb45f7328522b863001f" exitCode=0 Oct 30 23:58:36.090793 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.090759 5073 generic.go:356] "Generic (PLEG): container finished" podID="29aab21424359483efbf6437e8de7b4f" containerID="64fa0e1f4c8ca38b40ce07d3fd07b2c198c93432eb49dc9750b69cec53fb95a5" exitCode=0 Oct 30 23:58:36.099628 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.099608 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 30 23:58:36.099726 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.099637 5073 generic.go:356] "Generic (PLEG): container finished" podID="5fa50726-3415-4c7f-82ba-e6eb5115267a" containerID="3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d" exitCode=255 Oct 30 23:58:36.130874 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.130857 5073 manager.go:324] Recovery completed Oct 30 23:58:36.150697 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.150672 5073 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:58:36.150697 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.150695 5073 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:58:36.150849 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.150716 5073 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:58:36.150994 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.150973 5073 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:58:36.151049 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.150996 5073 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:58:36.151049 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151022 5073 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:58:36.151049 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151030 5073 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:58:36.151049 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151038 5073 policy_none.go:50] "Start" Oct 30 23:58:36.151212 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151058 5073 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:58:36.151212 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151070 5073 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:58:36.151923 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151904 5073 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:58:36.151923 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151924 5073 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:58:36.152023 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.151934 5073 policy_none.go:44] "Start" Oct 30 23:58:36.152352 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.152326 5073 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:58:36.156484 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.156457 5073 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:58:36.159956 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.159942 5073 manager.go:341] "Starting Device Plugin manager" Oct 30 23:58:36.160039 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.159968 5073 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:58:36.160039 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.159991 5073 server.go:87] "Starting device plugin registration server" Oct 30 23:58:36.160337 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160324 5073 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:58:36.160384 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160340 5073 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:58:36.160446 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160426 5073 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:58:36.160620 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160581 5073 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:58:36.160720 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160643 5073 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:58:36.160720 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160650 5073 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:58:36.160892 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.160722 5073 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:58:36.160631418Z","UUID":"43759086-e501-474f-a0df-d5a3405c0768","Handler":null,"Name":"","Endpoint":""} Oct 30 23:58:36.162153 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.162132 5073 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:58:36.162338 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.162312 5073 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:58:36.162397 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.162347 5073 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:58:36.261161 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.261137 5073 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:36.264937 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.264908 5073 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:36.265240 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.265224 5073 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:36.265279 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.265262 5073 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-246.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:36.265326 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.265317 5073 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.280941 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.280916 5073 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.281036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.280980 5073 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.353036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.352959 5073 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal"] Oct 30 23:58:36.353567 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353537 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"fc47763c26c267f0abea2ce914234e24","Type":"ContainerStarted","Data":"9f12bae981c0cc607b3dc403511c5657c8967fd380328205e04ad2c5b998e1f6"} Oct 30 23:58:36.353625 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353578 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"fc47763c26c267f0abea2ce914234e24","Type":"ContainerStarted","Data":"1ec79813e222652e3a6654d7dfb79f42bb8bff5ce02b8a01f5d269d328622b9a"} Oct 30 23:58:36.353625 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353623 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"29aab21424359483efbf6437e8de7b4f","Type":"ContainerStarted","Data":"703ec8608bc6d0e26513bda7e71f6cf7c16ec9cb45f1acff5611477115891841"} Oct 30 23:58:36.353697 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353561 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.353697 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353639 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"29aab21424359483efbf6437e8de7b4f","Type":"ContainerDied","Data":"64fa0e1f4c8ca38b40ce07d3fd07b2c198c93432eb49dc9750b69cec53fb95a5"} Oct 30 23:58:36.353697 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353656 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" event={"ID":"29aab21424359483efbf6437e8de7b4f","Type":"ContainerStarted","Data":"39a3e095c704f0a3277edd2bce469715b2d67aaa3e4b6a32302b3c296c08694d"} Oct 30 23:58:36.353823 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.353776 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.389157 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.389139 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:36.389228 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.389170 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:36.389228 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.389218 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.389314 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:36.389176 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.451483 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.451461 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.451568 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.451486 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.451568 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.451503 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc47763c26c267f0abea2ce914234e24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"fc47763c26c267f0abea2ce914234e24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.552427 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.552400 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc47763c26c267f0abea2ce914234e24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"fc47763c26c267f0abea2ce914234e24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.552520 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.552361 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc47763c26c267f0abea2ce914234e24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"fc47763c26c267f0abea2ce914234e24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.552520 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.552469 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.552520 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.552486 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.552613 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.552520 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:36.552613 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:36.552559 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29aab21424359483efbf6437e8de7b4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" (UID: \"29aab21424359483efbf6437e8de7b4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:37.011123 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.011101 5073 apiserver.go:52] "Watching apiserver" Oct 30 23:58:37.019985 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.019966 5073 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:58:37.020704 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.020680 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tz88x","openshift-network-console/networking-console-plugin-86c478bb88-46klc","openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz","kube-system/kas-connection-checker-685895c7c9-52d97","openshift-insights/periodic-gathering-bbltk-htq8x","openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc","openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq","openshift-network-diagnostics/network-check-target-6gwvc","kube-system/kas-connection-checker-685895c7c9-z49dz","openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps","openshift-ingress-canary/ingress-canary-nl6l6","openshift-multus/multus-wk7mn","openshift-console-operator/console-operator-7cd56b8dbc-sm45j","openshift-ovn-kubernetes/ovnkube-node-w75k7","openshift-image-registry/node-ca-n28vw","openshift-multus/network-metrics-daemon-dzqgd","openshift-image-registry/image-registry-69cf5d76f-vwpgd","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc","kube-system/global-pull-secret-syncer-qkj2v","openshift-dns/node-resolver-cjnpt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-jm9dj","openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4","openshift-service-ca/service-ca-5fffc97bdf-kxzdr","kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4qbw4","openshift-insights/insights-operator-7c4986d449-nnh9c","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc","openshift-dns/dns-default-5bnkm","openshift-multus/multus-additional-cni-plugins-2fpcf","openshift-image-registry/image-registry-59457478cb-rqvwg","openshift-ingress/router-default-f8b974b69-pvb5d","kube-system/konnectivity-agent-v79mw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc","openshift-insights/insights-runtime-extractor-jvn2x"] Oct 30 23:58:37.022043 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.022027 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.022647 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.022629 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.023482 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.023465 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.024693 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.024678 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.026078 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.026064 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:37.028455 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.028227 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.028455 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.028263 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.030157 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.030133 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 23:58:37.030577 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.030562 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.030626 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.030580 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 23:58:37.031108 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.031090 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.031167 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.031150 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.032167 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.032148 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 23:58:37.032759 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.032738 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 23:58:37.033817 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.033158 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 23:58:37.033817 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.033286 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:58:37.033817 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.033393 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 23:58:37.034421 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.034326 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 23:58:37.034421 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.034412 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 23:58:37.034668 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.034651 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fwwrl\"" Oct 30 23:58:37.034957 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.034940 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.035631 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035608 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.035733 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035637 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:37.035733 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035684 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.035733 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035687 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 23:58:37.035733 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035717 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.035941 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035745 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.035941 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035757 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.035941 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.035777 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 23:58:37.036885 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.036862 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.037015 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.036914 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 23:58:37.037015 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.036950 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 23:58:37.037015 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.036989 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.037015 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.036991 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 23:58:37.037221 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037009 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.037221 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037044 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.037221 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037045 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.037298 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037243 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqcpl\"" Oct 30 23:58:37.037329 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037313 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 23:58:37.037628 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037614 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:58:37.037842 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.037819 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 23:58:37.038031 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.038009 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 23:58:37.038031 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.038010 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.038170 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.038107 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 23:58:37.038170 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.038118 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.039173 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039153 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 23:58:37.039378 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039360 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 23:58:37.039455 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039407 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.039592 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039565 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.039674 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039657 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.039762 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039747 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.039932 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039916 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 23:58:37.039978 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039935 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:58:37.040070 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.040055 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.040154 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.040137 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:58:37.040207 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039664 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:58:37.040207 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.040168 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 23:58:37.040300 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.039755 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.040429 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.040415 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.040638 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.040623 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 23:58:37.040876 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.040861 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.043999 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.043984 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.054164 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054136 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvzk\" (UniqueName: \"kubernetes.io/projected/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-kube-api-access-trvzk\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:37.054164 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054169 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6afa2442-496b-45d4-ae52-133e0d21a60f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:37.054348 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054185 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.054348 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054206 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa50726-3415-4c7f-82ba-e6eb5115267a-serving-cert\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.054348 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054257 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c479d0-cb02-4549-acb7-3a206eb2ecb7-host\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.054348 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054294 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-kubelet\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.054348 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054327 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.054585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054381 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fxt\" (UniqueName: \"kubernetes.io/projected/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-kube-api-access-q9fxt\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.054585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054413 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-ovn\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.054585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054522 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-kubelet\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.054585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054533 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fa50726-3415-4c7f-82ba-e6eb5115267a-serving-cert\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.054585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054578 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-hostroot\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054619 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-daemon-config\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054646 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8p8\" (UniqueName: \"kubernetes.io/projected/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-kube-api-access-lr8p8\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054652 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/106281ea-1622-4966-bfd3-bdba5d770f5f-original-pull-secret\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054661 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6afa2442-496b-45d4-ae52-133e0d21a60f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054670 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzqk\" (UniqueName: \"kubernetes.io/projected/6df4d978-5bb3-48e6-a69f-519f0d85e465-kube-api-access-wjzqk\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054701 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5tn\" (UniqueName: \"kubernetes.io/projected/5fa50726-3415-4c7f-82ba-e6eb5115267a-kube-api-access-vq5tn\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054730 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.054831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054815 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-node-log\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.055139 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054845 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-system-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.055139 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054958 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lcv\" (UniqueName: \"kubernetes.io/projected/3ed58e8f-b3ed-4c94-aebf-0676c621799d-kube-api-access-26lcv\") pod \"kas-connection-checker-685895c7c9-z49dz\" (UID: \"3ed58e8f-b3ed-4c94-aebf-0676c621799d\") " pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:37.055139 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.054993 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjjd\" (UniqueName: \"kubernetes.io/projected/beff8893-2560-45f8-a96d-bce1e51dcf80-kube-api-access-5jjjd\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.055139 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055018 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.055139 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055055 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c-kube-api-access-vdl2z\") pod \"network-check-source-6f78d59f99-mc2zc\" (UID: \"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:37.055139 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055086 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055160 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b613d66f-e947-427e-9694-3f0b609f35c4-tmp-dir\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055185 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99-kube-api-access-jgfc9\") pod \"kas-connection-checker-685895c7c9-tz88x\" (UID: \"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99\") " pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055265 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b613d66f-e947-427e-9694-3f0b609f35c4-tmp-dir\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055280 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055347 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa2442-496b-45d4-ae52-133e0d21a60f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055383 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cnibin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.055432 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055403 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-socket-dir-parent\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-tmp\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055487 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/beff8893-2560-45f8-a96d-bce1e51dcf80-serving-cert\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055524 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055551 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b613d66f-e947-427e-9694-3f0b609f35c4-hosts-file\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055564 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-tmp\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055607 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6afa2442-496b-45d4-ae52-133e0d21a60f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055635 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-dbus\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.055744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055743 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/beff8893-2560-45f8-a96d-bce1e51dcf80-serving-cert\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.057099 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.055691 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-cnibin\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.057179 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057142 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-bin\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.057232 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057194 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cni-binary-copy\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.057232 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057222 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-netns\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.057355 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057281 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.057502 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057467 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71c479d0-cb02-4549-acb7-3a206eb2ecb7-serviceca\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.057609 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057537 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-env-overrides\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.057609 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057574 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-script-lib\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.057717 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057615 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-conf-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.057717 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057644 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-multus-certs\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.057717 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057678 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.057717 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057687 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-trusted-ca\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.057717 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057708 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db5e05a-532e-4937-a241-3a995c5c43ae-ovn-node-metrics-cert\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.057956 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057733 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cni-binary-copy\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.057956 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057741 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2f8fec0-412d-40ee-9d94-db3fbe04a651-nginx-conf\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:37.057956 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057776 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.057956 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057895 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/6afa2442-496b-45d4-ae52-133e0d21a60f-kube-api-access-wg79k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:37.057956 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057938 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-netns\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.058173 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.057995 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.058173 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058040 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2f8fec0-412d-40ee-9d94-db3fbe04a651-nginx-conf\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:37.058173 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058063 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbt5\" (UniqueName: \"kubernetes.io/projected/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-kube-api-access-9kbt5\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.058173 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058098 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-var-lib-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.058354 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058224 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.058354 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058256 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-k8s-cni-cncf-io\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.058354 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058291 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-stats-auth\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.058354 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058319 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.058526 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058366 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.058526 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058403 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.058526 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058440 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-slash\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.058526 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058468 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.058526 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058499 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng4p\" (UniqueName: \"kubernetes.io/projected/3db5e05a-532e-4937-a241-3a995c5c43ae-kube-api-access-wng4p\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.058715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058532 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-multus\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.058715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058585 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71c479d0-cb02-4549-acb7-3a206eb2ecb7-serviceca\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.058715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058627 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-service-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.058715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058661 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.058715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058695 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.058983 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058727 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.058983 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058775 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.058983 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058832 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.058983 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058864 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.058983 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058895 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kxc\" (UniqueName: \"kubernetes.io/projected/93866d6e-e48c-4e7f-823d-771497d4694b-kube-api-access-x8kxc\") pod \"kas-connection-checker-685895c7c9-52d97\" (UID: \"93866d6e-e48c-4e7f-823d-771497d4694b\") " pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:37.058983 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.058926 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6sbf\" (UniqueName: \"kubernetes.io/projected/fbea15b0-87b2-480b-9bb7-496f71990632-kube-api-access-j6sbf\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.059443 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.059419 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-stats-auth\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.059647 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.059630 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.059972 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.059951 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.060056 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.059977 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.060056 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060026 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-systemd-units\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.060155 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060055 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.060390 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060367 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.060451 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060434 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-netd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.060501 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060469 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-etc-kubernetes\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.060548 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060530 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beff8893-2560-45f8-a96d-bce1e51dcf80-service-ca-bundle\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.060593 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060532 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-default-certificate\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.060638 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060586 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-snapshots\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.060683 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060641 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060743 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-system-cni-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060815 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-log-socket\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060828 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-default-certificate\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060879 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-config\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.060959 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-bin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061156 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdchj\" (UniqueName: \"kubernetes.io/projected/b613d66f-e947-427e-9694-3f0b609f35c4-kube-api-access-jdchj\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061185 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-config\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061199 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/beff8893-2560-45f8-a96d-bce1e51dcf80-snapshots\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061217 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-systemd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061646 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061695 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-etc-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061721 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.061822 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061732 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-config\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061876 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061922 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-config\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061923 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-kubelet-config\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061989 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.061984 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5fa50726-3415-4c7f-82ba-e6eb5115267a-config\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.062036 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.062129 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgdj\" (UniqueName: \"kubernetes.io/projected/71c479d0-cb02-4549-acb7-3a206eb2ecb7-kube-api-access-9pgdj\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.062188 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxcg\" (UniqueName: \"kubernetes.io/projected/b470b4c4-501f-4d51-a04d-37f1f3d71c06-kube-api-access-ztxcg\") pod \"volume-data-source-validator-58c9f5f46d-px2hc\" (UID: \"b470b4c4-501f-4d51-a04d-37f1f3d71c06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.062269 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.062542 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.062425 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-serving-cert\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:37.062902 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.062882 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-serving-cert\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:37.066448 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.066432 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fxt\" (UniqueName: \"kubernetes.io/projected/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-kube-api-access-q9fxt\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.066516 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.066500 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzqk\" (UniqueName: \"kubernetes.io/projected/6df4d978-5bb3-48e6-a69f-519f0d85e465-kube-api-access-wjzqk\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.066547 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.066508 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvzk\" (UniqueName: \"kubernetes.io/projected/a2587ab9-e842-47c6-87b5-fb0b1ab8b76e-kube-api-access-trvzk\") pod \"service-ca-operator-544d749c8c-kz5pz\" (UID: \"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" Oct 30 23:58:37.068341 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.068313 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lcv\" (UniqueName: \"kubernetes.io/projected/3ed58e8f-b3ed-4c94-aebf-0676c621799d-kube-api-access-26lcv\") pod \"kas-connection-checker-685895c7c9-z49dz\" (UID: \"3ed58e8f-b3ed-4c94-aebf-0676c621799d\") " pod="kube-system/kas-connection-checker-685895c7c9-z49dz" Oct 30 23:58:37.068496 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.068371 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8p8\" (UniqueName: \"kubernetes.io/projected/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-kube-api-access-lr8p8\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.068885 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.068852 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c-kube-api-access-vdl2z\") pod \"network-check-source-6f78d59f99-mc2zc\" (UID: \"8ffadb0a-22ce-4aa0-aa04-d9f9a7c1676c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mc2zc" Oct 30 23:58:37.068982 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.068924 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5tn\" (UniqueName: \"kubernetes.io/projected/5fa50726-3415-4c7f-82ba-e6eb5115267a-kube-api-access-vq5tn\") pod \"console-operator-7cd56b8dbc-sm45j\" (UID: \"5fa50726-3415-4c7f-82ba-e6eb5115267a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:37.069068 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.068984 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.069068 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.069061 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjjd\" (UniqueName: \"kubernetes.io/projected/beff8893-2560-45f8-a96d-bce1e51dcf80-kube-api-access-5jjjd\") pod \"insights-operator-7c4986d449-nnh9c\" (UID: \"beff8893-2560-45f8-a96d-bce1e51dcf80\") " pod="openshift-insights/insights-operator-7c4986d449-nnh9c" Oct 30 23:58:37.069381 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.069353 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/3da2a0da-b9b7-41f0-8e8d-6e07faa06d99-kube-api-access-jgfc9\") pod \"kas-connection-checker-685895c7c9-tz88x\" (UID: \"3da2a0da-b9b7-41f0-8e8d-6e07faa06d99\") " pod="kube-system/kas-connection-checker-685895c7c9-tz88x" Oct 30 23:58:37.073678 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.073660 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbt5\" (UniqueName: \"kubernetes.io/projected/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-kube-api-access-9kbt5\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.073770 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.073756 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/6afa2442-496b-45d4-ae52-133e0d21a60f-kube-api-access-wg79k\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc\" (UID: \"6afa2442-496b-45d4-ae52-133e0d21a60f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" Oct 30 23:58:37.073885 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.073867 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kxc\" (UniqueName: \"kubernetes.io/projected/93866d6e-e48c-4e7f-823d-771497d4694b-kube-api-access-x8kxc\") pod \"kas-connection-checker-685895c7c9-52d97\" (UID: \"93866d6e-e48c-4e7f-823d-771497d4694b\") " pod="kube-system/kas-connection-checker-685895c7c9-52d97" Oct 30 23:58:37.074079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.074066 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgdj\" (UniqueName: \"kubernetes.io/projected/71c479d0-cb02-4549-acb7-3a206eb2ecb7-kube-api-access-9pgdj\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.076074 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.076053 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxcg\" (UniqueName: \"kubernetes.io/projected/b470b4c4-501f-4d51-a04d-37f1f3d71c06-kube-api-access-ztxcg\") pod \"volume-data-source-validator-58c9f5f46d-px2hc\" (UID: \"b470b4c4-501f-4d51-a04d-37f1f3d71c06\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-px2hc" Oct 30 23:58:37.076247 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.076231 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.076283 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.076259 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6sbf\" (UniqueName: \"kubernetes.io/projected/fbea15b0-87b2-480b-9bb7-496f71990632-kube-api-access-j6sbf\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.113468 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.113453 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:37.113676 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.113661 5073 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:37.124718 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.124700 5073 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:58:37.125068 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.125053 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:37.125136 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.125114 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:37.126501 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126484 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.126570 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126508 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:58:37.126570 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126524 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.126570 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126530 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.126665 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126656 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:58:37.126944 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126919 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126964 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126973 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.126995 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127005 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127014 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127019 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127024 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:58:37.127032 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127010 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127007 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127052 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127062 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127096 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127167 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127184 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127204 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127207 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127215 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127231 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127254 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:58:37.127391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.127290 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:58:37.128016 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.128000 5073 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:37.128104 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.128041 5073 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-246.us-east-2.compute.internal" Oct 30 23:58:37.128104 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.128064 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-env-overrides\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.128416 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.128395 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-script-lib\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.130691 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130621 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:58:37.130691 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130645 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:58:37.130967 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130858 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:58:37.130967 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130891 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 23:58:37.130967 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130903 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:58:37.130967 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130905 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 23:58:37.130967 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130916 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5dm5b\"" Oct 30 23:58:37.130967 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.130965 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 23:58:37.131196 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131034 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r8kbq\"" Oct 30 23:58:37.131196 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131041 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:58:37.131196 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131060 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:58:37.131196 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131080 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6ff8s\"" Oct 30 23:58:37.131461 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131448 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db5e05a-532e-4937-a241-3a995c5c43ae-ovnkube-config\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.131656 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131640 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7b2l\"" Oct 30 23:58:37.131656 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131640 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j47x2\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131678 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dtg7z\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131679 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131702 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g645g\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131684 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6l954\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131713 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpmk\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131715 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxfk8\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131728 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vmg6c\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131716 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ccbg\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131747 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrv6t\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131751 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131761 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:58:37.131831 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131804 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72z8g\"" Oct 30 23:58:37.132150 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.131892 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpqng\"" Oct 30 23:58:37.132328 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.132311 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbea15b0-87b2-480b-9bb7-496f71990632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.133670 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.133655 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng4p\" (UniqueName: \"kubernetes.io/projected/3db5e05a-532e-4937-a241-3a995c5c43ae-kube-api-access-wng4p\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.135204 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.135189 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-daemon-config\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.136531 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.136516 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.136768 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.136753 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdchj\" (UniqueName: \"kubernetes.io/projected/b613d66f-e947-427e-9694-3f0b609f35c4-kube-api-access-jdchj\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.138661 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.138642 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db5e05a-532e-4937-a241-3a995c5c43ae-ovn-node-metrics-cert\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.162676 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162658 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.162755 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162682 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2528eea-3e1e-49eb-ad02-537965818be1-signing-cabundle\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:37.162850 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162831 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-sys\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.162907 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162864 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.162907 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162879 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b2528eea-3e1e-49eb-ad02-537965818be1-signing-cabundle\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:37.162994 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162905 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.162994 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162948 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.163075 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.162999 5073 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:58:37.163075 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163030 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.163075 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163058 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-netd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163112 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163124 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-netd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163143 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-systemd\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163163 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-bin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163179 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163184 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.163214 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163193 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163222 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-bin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163231 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-etc-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163271 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163279 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-etc-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163273 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.163344 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:37.163475 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.163458 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:37.663444485 +0000 UTC m=+1.724219172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163564 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-kubelet-config\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163595 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq72r\" (UniqueName: \"kubernetes.io/projected/686b1996-b6f2-44e9-b9bd-8acb156e833d-kube-api-access-zq72r\") pod \"migrator-66df5d496b-jwqps\" (UID: \"686b1996-b6f2-44e9-b9bd-8acb156e833d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163624 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-kubelet\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163629 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-kubelet-config\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163646 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-hostroot\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163672 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163679 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-kubelet\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163724 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-hostroot\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163728 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sgf\" (UniqueName: \"kubernetes.io/projected/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kube-api-access-l7sgf\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163756 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163808 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c199e89-40d7-4a83-815c-f3533fcc31c4-iptables-alerter-script\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163839 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163865 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-lib-modules\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163890 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-dbus\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163916 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-bin\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.163809 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163940 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-netns\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163953 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.163966 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-conf-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164006 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-multus-certs\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164017 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164034 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164057 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27ea3-18d4-4aff-a92b-a4290ec72714-config-volume\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164078 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-netns\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164101 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-registration-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164124 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-var-lib-kubelet\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164172 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-var-lib-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164195 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164218 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-k8s-cni-cncf-io\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164247 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164273 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysconfig\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164296 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-run\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164308 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c199e89-40d7-4a83-815c-f3533fcc31c4-iptables-alerter-script\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164320 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-tmp\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164324 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164345 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb57k\" (UniqueName: \"kubernetes.io/projected/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-kube-api-access-pb57k\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164483 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-cni-bin\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164489 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-netns\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164489 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164524 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164543 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/106281ea-1622-4966-bfd3-bdba5d770f5f-dbus\") pod \"global-pull-secret-syncer-qkj2v\" (UID: \"106281ea-1622-4966-bfd3-bdba5d770f5f\") " pod="kube-system/global-pull-secret-syncer-qkj2v" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164559 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86e12f20-75a3-480a-81b1-f6bf080a6733-agent-certs\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164602 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-k8s-cni-cncf-io\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164605 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-systemd-units\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164631 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-etc-kubernetes\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164663 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-sys-fs\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164692 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164760 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwwx\" (UniqueName: \"kubernetes.io/projected/f270b734-9691-480f-8d55-1448c307a69b-kube-api-access-8fwwx\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164809 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-system-cni-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164834 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-log-socket\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.164854 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164862 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164889 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-conf\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86e12f20-75a3-480a-81b1-f6bf080a6733-agent-certs\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164916 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-systemd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164936 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-multus-certs\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164943 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.164973 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165011 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165038 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a27ea3-18d4-4aff-a92b-a4290ec72714-tmp-dir\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165069 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165087 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165196 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c479d0-cb02-4549-acb7-3a206eb2ecb7-host\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165231 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-kubelet\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165260 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-etc-tuned\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165282 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165306 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-ovn\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165329 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-data-volume\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165356 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86e12f20-75a3-480a-81b1-f6bf080a6733-konnectivity-ca\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165383 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5l2\" (UniqueName: \"kubernetes.io/projected/d0ffac65-ad93-4617-afbd-9d2fe6153391-kube-api-access-wb5l2\") pod \"migrator-66df5d496b-wmdc4\" (UID: \"d0ffac65-ad93-4617-afbd-9d2fe6153391\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165408 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4m8w\" (UniqueName: \"kubernetes.io/projected/4c199e89-40d7-4a83-815c-f3533fcc31c4-kube-api-access-x4m8w\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165431 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-node-log\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165471 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-system-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165504 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165528 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c199e89-40d7-4a83-815c-f3533fcc31c4-host-slash\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165554 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165578 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cnibin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165617 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-socket-dir-parent\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165643 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b613d66f-e947-427e-9694-3f0b609f35c4-hosts-file\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165666 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-api-access-kcnkw\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165715 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165860 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-cnibin\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165890 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z2d\" (UniqueName: \"kubernetes.io/projected/e3036952-ce27-471a-9fa5-933ea133e22e-kube-api-access-67z2d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165902 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-node-log\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165920 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165947 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2528eea-3e1e-49eb-ad02-537965818be1-signing-key\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165962 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-system-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.165974 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-kubernetes\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166006 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f270b734-9691-480f-8d55-1448c307a69b-cert\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166020 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-log-socket\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166039 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpfp\" (UniqueName: \"kubernetes.io/projected/b2528eea-3e1e-49eb-ad02-537965818be1-kube-api-access-prpfp\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166052 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-tmp\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166071 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-crio-socket\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166104 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166118 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-data-volume\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166165 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-systemd\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166182 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-var-lib-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166189 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"image-registry-69cf5d76f-vwpgd\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166255 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-etc-kubernetes\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166298 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-run-netns\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166329 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86e12f20-75a3-480a-81b1-f6bf080a6733-konnectivity-ca\") pod \"konnectivity-agent-v79mw\" (UID: \"86e12f20-75a3-480a-81b1-f6bf080a6733\") " pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166341 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-systemd-units\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166394 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-kubelet\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166432 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3036952-ce27-471a-9fa5-933ea133e22e-etc-tuned\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166475 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a27ea3-18d4-4aff-a92b-a4290ec72714-config-volume\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.166511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166508 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-ovn\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166572 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-cnibin\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166596 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbea15b0-87b2-480b-9bb7-496f71990632-system-cni-dir\") pod \"multus-additional-cni-plugins-2fpcf\" (UID: \"fbea15b0-87b2-480b-9bb7-496f71990632\") " pod="openshift-multus/multus-additional-cni-plugins-2fpcf" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166601 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-cnibin\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-conf-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166656 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166668 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b613d66f-e947-427e-9694-3f0b609f35c4-hosts-file\") pod \"node-resolver-cjnpt\" (UID: \"b613d66f-e947-427e-9694-3f0b609f35c4\") " pod="openshift-dns/node-resolver-cjnpt" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166659 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-socket-dir-parent\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166691 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a27ea3-18d4-4aff-a92b-a4290ec72714-tmp-dir\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166001 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-pz8mc\" (UID: \"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166768 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c479d0-cb02-4549-acb7-3a206eb2ecb7-host\") pod \"node-ca-n28vw\" (UID: \"71c479d0-cb02-4549-acb7-3a206eb2ecb7\") " pod="openshift-image-registry/node-ca-n28vw" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166855 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.166862 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-multus-cni-dir\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.167001 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:37.666984072 +0000 UTC m=+1.727758746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167044 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-socket-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167073 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-device-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167098 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-modprobe-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167122 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167145 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-host\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167333 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkh6\" (UniqueName: \"kubernetes.io/projected/e1a27ea3-18d4-4aff-a92b-a4290ec72714-kube-api-access-6jkh6\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167372 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-slash\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167373 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df4d978-5bb3-48e6-a69f-519f0d85e465-metrics-certs\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167414 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167425 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167445 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f270b734-9691-480f-8d55-1448c307a69b-cert\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167446 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-multus\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167492 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-run-openvswitch\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbbbfcf-c966-49b9-8cf3-91b77469bd2b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-whqjq\" (UID: \"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167554 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a607b81-bbfa-4a4e-9828-2065d1e3bf49-host-var-lib-cni-multus\") pod \"multus-wk7mn\" (UID: \"8a607b81-bbfa-4a4e-9828-2065d1e3bf49\") " pod="openshift-multus/multus-wk7mn" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167574 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db5e05a-532e-4937-a241-3a995c5c43ae-host-slash\") pod \"ovnkube-node-w75k7\" (UID: \"3db5e05a-532e-4937-a241-3a995c5c43ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:37.168503 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.167687 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b2528eea-3e1e-49eb-ad02-537965818be1-signing-key\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:37.178125 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.178098 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sgf\" (UniqueName: \"kubernetes.io/projected/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kube-api-access-l7sgf\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.178219 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.178134 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq72r\" (UniqueName: \"kubernetes.io/projected/686b1996-b6f2-44e9-b9bd-8acb156e833d-kube-api-access-zq72r\") pod \"migrator-66df5d496b-jwqps\" (UID: \"686b1996-b6f2-44e9-b9bd-8acb156e833d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jwqps" Oct 30 23:58:37.179742 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.179721 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.179841 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.179778 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") pod \"periodic-gathering-bbltk-htq8x\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:58:37.181698 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181325 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5l2\" (UniqueName: \"kubernetes.io/projected/d0ffac65-ad93-4617-afbd-9d2fe6153391-kube-api-access-wb5l2\") pod \"migrator-66df5d496b-wmdc4\" (UID: \"d0ffac65-ad93-4617-afbd-9d2fe6153391\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wmdc4" Oct 30 23:58:37.181698 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181588 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4m8w\" (UniqueName: \"kubernetes.io/projected/4c199e89-40d7-4a83-815c-f3533fcc31c4-kube-api-access-x4m8w\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.181698 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181659 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnkw\" (UniqueName: \"kubernetes.io/projected/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-api-access-kcnkw\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.181904 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181717 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwwx\" (UniqueName: \"kubernetes.io/projected/f270b734-9691-480f-8d55-1448c307a69b-kube-api-access-8fwwx\") pod \"ingress-canary-nl6l6\" (UID: \"f270b734-9691-480f-8d55-1448c307a69b\") " pod="openshift-ingress-canary/ingress-canary-nl6l6" Oct 30 23:58:37.181904 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181851 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx55\" (UniqueName: \"kubernetes.io/projected/b31374f9-c1aa-4288-a5db-c88ec5b3f732-kube-api-access-xvx55\") pod \"network-check-target-6gwvc\" (UID: \"b31374f9-c1aa-4288-a5db-c88ec5b3f732\") " pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:37.181904 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181884 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z2d\" (UniqueName: \"kubernetes.io/projected/e3036952-ce27-471a-9fa5-933ea133e22e-kube-api-access-67z2d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.181904 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181895 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.182121 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.181961 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpfp\" (UniqueName: \"kubernetes.io/projected/b2528eea-3e1e-49eb-ad02-537965818be1-kube-api-access-prpfp\") pod \"service-ca-5fffc97bdf-kxzdr\" (UID: \"b2528eea-3e1e-49eb-ad02-537965818be1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-kxzdr" Oct 30 23:58:37.183016 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.182996 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb57k\" (UniqueName: \"kubernetes.io/projected/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-kube-api-access-pb57k\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:37.183496 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.183478 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkh6\" (UniqueName: \"kubernetes.io/projected/e1a27ea3-18d4-4aff-a92b-a4290ec72714-kube-api-access-6jkh6\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.268669 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268603 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.268669 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268635 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-sys-fs\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.268669 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268651 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.268669 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268667 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-conf\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268690 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268738 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-sys-fs\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268738 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268762 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c199e89-40d7-4a83-815c-f3533fcc31c4-host-slash\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268744 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268818 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c199e89-40d7-4a83-815c-f3533fcc31c4-host-slash\") pod \"iptables-alerter-jm9dj\" (UID: \"4c199e89-40d7-4a83-815c-f3533fcc31c4\") " pod="openshift-network-operator/iptables-alerter-jm9dj" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268867 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-conf\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268818 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:37.268929 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.268882 5073 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.268942 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:37.768924215 +0000 UTC m=+1.829698902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : secret "metrics-daemon-secret" not found Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268956 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-kubernetes\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268975 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-crio-socket\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.268996 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269022 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-socket-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269029 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-crio-socket\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269033 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-kubernetes\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269044 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-device-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269067 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-modprobe-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269082 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269096 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-host\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269105 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-socket-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269153 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-sys\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269157 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-device-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269176 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269201 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-systemd\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269207 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-sys\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269221 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-modprobe-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269228 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-host\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269255 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysctl-d\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269261 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.269256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269265 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-systemd\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.269324 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.269365 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:37.769350751 +0000 UTC m=+1.830125438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269378 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-lib-modules\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269399 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-registration-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269419 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-var-lib-kubelet\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269476 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysconfig\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269497 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-run\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269498 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-lib-modules\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e72e0f4-7067-409f-bc3a-0dcff8e656fa-registration-dir\") pod \"aws-ebs-csi-driver-node-696hc\" (UID: \"1e72e0f4-7067-409f-bc3a-0dcff8e656fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-696hc" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269564 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-run\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269569 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269579 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-etc-sysconfig\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.270044 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.269609 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3036952-ce27-471a-9fa5-933ea133e22e-var-lib-kubelet\") pod \"tuned-4qbw4\" (UID: \"e3036952-ce27-471a-9fa5-933ea133e22e\") " pod="openshift-cluster-node-tuning-operator/tuned-4qbw4" Oct 30 23:58:37.271268 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.271247 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"image-registry-59457478cb-rqvwg\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.271417 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.271401 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a27ea3-18d4-4aff-a92b-a4290ec72714-metrics-tls\") pod \"dns-default-5bnkm\" (UID: \"e1a27ea3-18d4-4aff-a92b-a4290ec72714\") " pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.322543 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.322526 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" Oct 30 23:58:37.323648 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.323634 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" Oct 30 23:58:37.324598 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.324586 5073 scope.go:122] "RemoveContainer" containerID="3e8cea670e4bf7dd74d5ba01c1373ed0f3f98e4b17b6ab24bf86b4529a34bf7d" Oct 30 23:58:37.324960 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.324941 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:37.340871 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.340855 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:37.341962 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.341943 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:37.629272 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.629243 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq"] Oct 30 23:58:37.633127 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:37.633101 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbbbfcf_c966_49b9_8cf3_91b77469bd2b.slice/crio-2caaaaaaa5eeae8abe0d47741f50c2dd22abf43f212519cd6cebec4e5dd314e2 WatchSource:0}: Error finding container 2caaaaaaa5eeae8abe0d47741f50c2dd22abf43f212519cd6cebec4e5dd314e2: Status 404 returned error can't find the container with id 2caaaaaaa5eeae8abe0d47741f50c2dd22abf43f212519cd6cebec4e5dd314e2 Oct 30 23:58:37.635613 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.635592 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:58:37.676207 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.676185 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:37.676305 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.676230 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:37.676353 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.676317 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:37.676389 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.676366 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:38.67635095 +0000 UTC m=+2.737125640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:37.676389 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.676381 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:38.676375137 +0000 UTC m=+2.737149809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:37.688117 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.688096 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc"] Oct 30 23:58:37.703757 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.703726 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bnkm"] Oct 30 23:58:37.707371 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:37.707347 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a27ea3_18d4_4aff_a92b_a4290ec72714.slice/crio-9cf2c1f31fcf63689bde654ab210e91a35f44ed9e673a492326455f18deb6353 WatchSource:0}: Error finding container 9cf2c1f31fcf63689bde654ab210e91a35f44ed9e673a492326455f18deb6353: Status 404 returned error can't find the container with id 9cf2c1f31fcf63689bde654ab210e91a35f44ed9e673a492326455f18deb6353 Oct 30 23:58:37.757671 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.757634 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69cf5d76f-vwpgd"] Oct 30 23:58:37.763747 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:37.763724 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500001f6_9b33_4347_b5f5_cfcf79e6ed09.slice/crio-eec32dd11618bffa442f1a3f5f515264e118b77cfb0c850e383dc57249230d1c WatchSource:0}: Error finding container eec32dd11618bffa442f1a3f5f515264e118b77cfb0c850e383dc57249230d1c: Status 404 returned error can't find the container with id eec32dd11618bffa442f1a3f5f515264e118b77cfb0c850e383dc57249230d1c Oct 30 23:58:37.770851 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.770826 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59457478cb-rqvwg"] Oct 30 23:58:37.777301 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:37.777274 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10737b2e_c98b_433d_a846_c087bfc5410f.slice/crio-94d4770c132787315ff6184d87286c2648a4a9d088a524a0161a4d7d0752e6e8 WatchSource:0}: Error finding container 94d4770c132787315ff6184d87286c2648a4a9d088a524a0161a4d7d0752e6e8: Status 404 returned error can't find the container with id 94d4770c132787315ff6184d87286c2648a4a9d088a524a0161a4d7d0752e6e8 Oct 30 23:58:37.777559 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.777537 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:37.777644 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:37.777628 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:37.777859 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.777842 5073 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:58:37.777931 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.777893 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:37.777993 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.777900 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs podName:96ddf74c-7e70-4e9d-b5fd-77203f8d2179 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:38.777882245 +0000 UTC m=+2.838656938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs") pod "network-metrics-daemon-dzqgd" (UID: "96ddf74c-7e70-4e9d-b5fd-77203f8d2179") : secret "metrics-daemon-secret" not found Oct 30 23:58:37.777993 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:37.777975 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:38.777960993 +0000 UTC m=+2.838735670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:38.121061 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.120983 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bnkm" event={"ID":"e1a27ea3-18d4-4aff-a92b-a4290ec72714","Type":"ContainerStarted","Data":"9cf2c1f31fcf63689bde654ab210e91a35f44ed9e673a492326455f18deb6353"} Oct 30 23:58:38.122856 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.122831 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" event={"ID":"10737b2e-c98b-433d-a846-c087bfc5410f","Type":"ContainerStarted","Data":"009cdd1c97200f312e2a61a43db65c160d0b1b360ed3072edc223d182187c6e1"} Oct 30 23:58:38.123020 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.122871 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" event={"ID":"10737b2e-c98b-433d-a846-c087bfc5410f","Type":"ContainerStarted","Data":"94d4770c132787315ff6184d87286c2648a4a9d088a524a0161a4d7d0752e6e8"} Oct 30 23:58:38.124508 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.124486 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" event={"ID":"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b","Type":"ContainerStarted","Data":"2caaaaaaa5eeae8abe0d47741f50c2dd22abf43f212519cd6cebec4e5dd314e2"} Oct 30 23:58:38.126608 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.126590 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 30 23:58:38.126716 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.126690 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" event={"ID":"5fa50726-3415-4c7f-82ba-e6eb5115267a","Type":"ContainerStarted","Data":"afe92515e7e55e7331e8ad6b6ed6b983dad0c776744e286af6c0fae44c519dd5"} Oct 30 23:58:38.131876 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.130359 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" event={"ID":"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad","Type":"ContainerStarted","Data":"37a7d9705ef728eecdf05e0e2b74f206dcfecaa9ef860ba10a238539fcbdd871"} Oct 30 23:58:38.133480 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.132185 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" event={"ID":"500001f6-9b33-4347-b5f5-cfcf79e6ed09","Type":"ContainerStarted","Data":"ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f"} Oct 30 23:58:38.133480 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.132235 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" event={"ID":"500001f6-9b33-4347-b5f5-cfcf79e6ed09","Type":"ContainerStarted","Data":"eec32dd11618bffa442f1a3f5f515264e118b77cfb0c850e383dc57249230d1c"} Oct 30 23:58:38.688539 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.688489 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:38.688749 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.688649 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:38.688749 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:38.688654 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:38.688995 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:38.688754 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle podName:6df4d978-5bb3-48e6-a69f-519f0d85e465 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:40.688733984 +0000 UTC m=+4.749508671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle") pod "router-default-f8b974b69-pvb5d" (UID: "6df4d978-5bb3-48e6-a69f-519f0d85e465") : configmap references non-existent config key: service-ca.crt Oct 30 23:58:38.688995 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:38.688798 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:40.688770414 +0000 UTC m=+4.749545091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:38.790165 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.790130 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:38.790337 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.790294 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:38.790480 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:38.790457 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:38.790555 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:38.790538 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:40.790518267 +0000 UTC m=+4.851292956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:38.792934 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.792914 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ddf74c-7e70-4e9d-b5fd-77203f8d2179-metrics-certs\") pod \"network-metrics-daemon-dzqgd\" (UID: \"96ddf74c-7e70-4e9d-b5fd-77203f8d2179\") " pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:38.837540 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:38.837508 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzqgd" Oct 30 23:58:39.130641 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:39.130604 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dzqgd"] Oct 30 23:58:39.142145 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:39.135852 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ddf74c_7e70_4e9d_b5fd_77203f8d2179.slice/crio-536fe2ce2b299136252f50834db33f7450cdc5158f0062313b88ef593cd4f66e WatchSource:0}: Error finding container 536fe2ce2b299136252f50834db33f7450cdc5158f0062313b88ef593cd4f66e: Status 404 returned error can't find the container with id 536fe2ce2b299136252f50834db33f7450cdc5158f0062313b88ef593cd4f66e Oct 30 23:58:39.142145 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:39.137060 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:39.142145 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:39.137597 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:39.167182 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:39.167163 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:39.228182 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:39.228151 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:58:40.144686 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.144602 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bnkm" event={"ID":"e1a27ea3-18d4-4aff-a92b-a4290ec72714","Type":"ContainerStarted","Data":"f42e8c12c710da03fa0cbce5e5a122d50543e6127cfca8cea0d82add134d8eee"} Oct 30 23:58:40.144686 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.144665 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bnkm" event={"ID":"e1a27ea3-18d4-4aff-a92b-a4290ec72714","Type":"ContainerStarted","Data":"0480b6f73bd3aabfa99e4479b016db4addb59c3455a51db15eab2f2ea4f66093"} Oct 30 23:58:40.145200 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.144855 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:40.146800 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.146758 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-whqjq" event={"ID":"7dbbbfcf-c966-49b9-8cf3-91b77469bd2b","Type":"ContainerStarted","Data":"f9a0ab2c31c67f6c574007307944c6ce2d7fd42e3c1fb194c0de8fd95e18f2e4"} Oct 30 23:58:40.154504 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.154473 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzqgd" event={"ID":"96ddf74c-7e70-4e9d-b5fd-77203f8d2179","Type":"ContainerStarted","Data":"536fe2ce2b299136252f50834db33f7450cdc5158f0062313b88ef593cd4f66e"} Oct 30 23:58:40.710413 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.710307 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:40.710904 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.710875 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:40.711046 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:40.711028 5073 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:58:40.711108 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.711062 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6df4d978-5bb3-48e6-a69f-519f0d85e465-service-ca-bundle\") pod \"router-default-f8b974b69-pvb5d\" (UID: \"6df4d978-5bb3-48e6-a69f-519f0d85e465\") " pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:40.711108 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:40.711101 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert podName:b2f8fec0-412d-40ee-9d94-db3fbe04a651 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:44.711083234 +0000 UTC m=+8.771857920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-46klc" (UID: "b2f8fec0-412d-40ee-9d94-db3fbe04a651") : secret "networking-console-plugin-cert" not found Oct 30 23:58:40.811596 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.811561 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:40.811769 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:40.811741 5073 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:58:40.811927 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:58:40.811839 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls podName:aeeea8e9-d19d-4518-b9cb-285b96d9a5d1 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:44.811821216 +0000 UTC m=+8.872595902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvn2x" (UID: "aeeea8e9-d19d-4518-b9cb-285b96d9a5d1") : secret "insights-runtime-extractor-tls" not found Oct 30 23:58:40.923158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.923129 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:40.945590 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:40.945568 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v79mw" Oct 30 23:58:41.160734 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.160702 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" event={"ID":"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad","Type":"ContainerStarted","Data":"5db59c2b090b93f98f9a41b51e5e2e24f016c678ca6a1f41bd1db6a704e4f460"} Oct 30 23:58:41.161196 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.160758 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-pz8mc" event={"ID":"6bd2b9e3-60fe-4a46-be1c-cb5fce3c05ad","Type":"ContainerStarted","Data":"c4d4b357aca579ed66845fed2b489c5fd62bde697d47d872aeb153fe282c4fcb"} Oct 30 23:58:41.162749 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.162720 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzqgd" event={"ID":"96ddf74c-7e70-4e9d-b5fd-77203f8d2179","Type":"ContainerStarted","Data":"4fb430a34d80bacd9ccbc830a4812f4da861f49f38dc7a127f5049edda369aa8"} Oct 30 23:58:41.162904 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.162769 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzqgd" event={"ID":"96ddf74c-7e70-4e9d-b5fd-77203f8d2179","Type":"ContainerStarted","Data":"0c8dd058e44f47b595b95e0286ef554cfb532670fff81c060f35a39a941e965e"} Oct 30 23:58:41.215622 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.215592 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f8b974b69-pvb5d"] Oct 30 23:58:41.219749 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:41.219726 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df4d978_5bb3_48e6_a69f_519f0d85e465.slice/crio-d4507451104b00520e51d2c8412093197fcac27434af425657b58151e366fc11 WatchSource:0}: Error finding container d4507451104b00520e51d2c8412093197fcac27434af425657b58151e366fc11: Status 404 returned error can't find the container with id d4507451104b00520e51d2c8412093197fcac27434af425657b58151e366fc11 Oct 30 23:58:41.850037 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.850005 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:41.850225 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.850136 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:41.853578 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:41.853560 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sm45j" Oct 30 23:58:42.167769 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:42.167662 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f8b974b69-pvb5d" event={"ID":"6df4d978-5bb3-48e6-a69f-519f0d85e465","Type":"ContainerStarted","Data":"bef6ffd1cf5c6f8fdb3461e701902ab8cde08c119188481bb7fbdb123eb2a580"} Oct 30 23:58:42.167769 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:42.167739 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f8b974b69-pvb5d" event={"ID":"6df4d978-5bb3-48e6-a69f-519f0d85e465","Type":"ContainerStarted","Data":"d4507451104b00520e51d2c8412093197fcac27434af425657b58151e366fc11"} Oct 30 23:58:42.219404 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:42.219376 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:58:42.923533 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:42.923502 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:42.926211 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:42.926178 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:43.042969 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:43.042939 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:58:44.175526 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.175499 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:44.741258 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.741225 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:44.743996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.743966 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2f8fec0-412d-40ee-9d94-db3fbe04a651-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-46klc\" (UID: \"b2f8fec0-412d-40ee-9d94-db3fbe04a651\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:44.775838 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.775815 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:44.776062 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.776048 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:44.826799 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.826765 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" Oct 30 23:58:44.842568 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.842539 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:44.844877 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:44.844854 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeeea8e9-d19d-4518-b9cb-285b96d9a5d1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvn2x\" (UID: \"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1\") " pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:45.099052 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:45.098978 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-46klc"] Oct 30 23:58:45.102569 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:45.102541 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f8fec0_412d_40ee_9d94_db3fbe04a651.slice/crio-7a4a4411058a3c590339c7686b62e2e6a4aa6936ce03243283aedf01012724ea WatchSource:0}: Error finding container 7a4a4411058a3c590339c7686b62e2e6a4aa6936ce03243283aedf01012724ea: Status 404 returned error can't find the container with id 7a4a4411058a3c590339c7686b62e2e6a4aa6936ce03243283aedf01012724ea Oct 30 23:58:45.145124 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:45.145103 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvn2x" Oct 30 23:58:45.187347 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:45.187314 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" event={"ID":"b2f8fec0-412d-40ee-9d94-db3fbe04a651","Type":"ContainerStarted","Data":"7a4a4411058a3c590339c7686b62e2e6a4aa6936ce03243283aedf01012724ea"} Oct 30 23:58:45.421928 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:45.421842 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvn2x"] Oct 30 23:58:45.427214 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:45.426590 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeeea8e9_d19d_4518_b9cb_285b96d9a5d1.slice/crio-9cb556c49531df72e74ad86abc0d1df63d616fb95233d27f135043f41e659c00 WatchSource:0}: Error finding container 9cb556c49531df72e74ad86abc0d1df63d616fb95233d27f135043f41e659c00: Status 404 returned error can't find the container with id 9cb556c49531df72e74ad86abc0d1df63d616fb95233d27f135043f41e659c00 Oct 30 23:58:46.192261 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:46.192235 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvn2x" event={"ID":"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1","Type":"ContainerStarted","Data":"33c2fc0a0a92a5ca437e0c6bd8c0c988cccc7ab1d6756ae623526514a019a7b3"} Oct 30 23:58:46.192645 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:46.192285 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvn2x" event={"ID":"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1","Type":"ContainerStarted","Data":"9cb556c49531df72e74ad86abc0d1df63d616fb95233d27f135043f41e659c00"} Oct 30 23:58:47.198393 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:47.198358 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-46klc" event={"ID":"b2f8fec0-412d-40ee-9d94-db3fbe04a651","Type":"ContainerStarted","Data":"78d48cf4442a762b053e59b824a08a392ecf84e9d29af062ef30ef2d98b9468e"} Oct 30 23:58:47.200709 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:47.200682 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvn2x" event={"ID":"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1","Type":"ContainerStarted","Data":"d1591eccffb1dbc8a09122e45ee70bbd4a1506f9ec93cf18a666dbf9319e17f4"} Oct 30 23:58:48.206876 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:48.206833 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvn2x" event={"ID":"aeeea8e9-d19d-4518-b9cb-285b96d9a5d1","Type":"ContainerStarted","Data":"194925bce8f93876c156832048189121f05c0f1e0514d7f2eb13aa0bf92386c6"} Oct 30 23:58:48.928538 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:48.928505 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:48.928679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:48.928627 5073 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:48.929778 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:48.929757 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f8b974b69-pvb5d" Oct 30 23:58:49.107702 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:49.107667 5073 patch_prober.go:29] interesting pod/image-registry-59457478cb-rqvwg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:58:49.107847 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:49.107715 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" podUID="10737b2e-c98b-433d-a846-c087bfc5410f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:58:49.404317 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:49.404290 5073 patch_prober.go:29] interesting pod/image-registry-69cf5d76f-vwpgd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:58:49.404661 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:49.404349 5073 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:58:50.164631 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:50.164604 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5bnkm" Oct 30 23:58:52.224053 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:52.224025 5073 patch_prober.go:29] interesting pod/image-registry-69cf5d76f-vwpgd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:58:52.224436 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:52.224076 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:58:53.144742 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.144712 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w75k7" Oct 30 23:58:53.413130 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.413050 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr"] Oct 30 23:58:53.443835 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.432228 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:53.444485 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.441372 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5vs22"] Oct 30 23:58:53.447963 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.447936 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c69847f79-f92bg"] Oct 30 23:58:53.448310 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.448293 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.450579 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.450542 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qhgvd\"" Oct 30 23:58:53.451769 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.451751 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr"] Oct 30 23:58:53.451889 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.451872 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.457592 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.455883 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5vs22"] Oct 30 23:58:53.458035 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.458017 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 23:58:53.458244 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.458218 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 23:58:53.458770 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.458739 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 23:58:53.464020 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.463775 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c69847f79-f92bg"] Oct 30 23:58:53.465224 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.465206 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69cf5d76f-vwpgd"] Oct 30 23:58:53.472844 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.472823 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 23:58:53.472948 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.472927 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 23:58:53.474382 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.474361 5073 patch_prober.go:29] interesting pod/image-registry-69cf5d76f-vwpgd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:58:53.474476 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.474399 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:58:53.506574 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.506550 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cef65b43-4a0c-4b34-a4e9-579f817084e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qgqzr\" (UID: \"cef65b43-4a0c-4b34-a4e9-579f817084e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:53.533556 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.533533 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd666768c-vdtlt"] Oct 30 23:58:53.536771 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.536735 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.550989 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.550966 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5jhf2\"" Oct 30 23:58:53.551124 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.551105 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd666768c-vdtlt"] Oct 30 23:58:53.551321 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.551299 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 23:58:53.551426 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.551332 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-txb26\"" Oct 30 23:58:53.551426 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.551341 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 23:58:53.551426 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.551420 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 23:58:53.607912 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.607886 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608021 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.607929 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608021 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.607958 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e412acfd-c25b-4802-8d58-d99dd3390098-registry-certificates\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608093 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608082 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5gs\" (UniqueName: \"kubernetes.io/projected/3ffa894e-03f3-4791-9afb-76d99ec77a24-kube-api-access-8l5gs\") pod \"downloads-6dd789c8b7-5vs22\" (UID: \"3ffa894e-03f3-4791-9afb-76d99ec77a24\") " pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.608128 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608101 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e412acfd-c25b-4802-8d58-d99dd3390098-installation-pull-secrets\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608128 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608122 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e412acfd-c25b-4802-8d58-d99dd3390098-image-registry-private-configuration\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608220 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608201 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cef65b43-4a0c-4b34-a4e9-579f817084e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qgqzr\" (UID: \"cef65b43-4a0c-4b34-a4e9-579f817084e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:53.608292 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608232 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608292 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608252 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-bound-sa-token\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608292 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608273 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5vm\" (UniqueName: \"kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608379 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608338 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608379 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608368 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ffa894e-03f3-4791-9afb-76d99ec77a24-tmp\") pod \"downloads-6dd789c8b7-5vs22\" (UID: \"3ffa894e-03f3-4791-9afb-76d99ec77a24\") " pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.608442 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608389 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608472 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608440 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.608504 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608473 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc86s\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-kube-api-access-zc86s\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608504 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608498 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-registry-tls\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608562 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608524 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e412acfd-c25b-4802-8d58-d99dd3390098-ca-trust-extracted\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.608612 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.608603 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e412acfd-c25b-4802-8d58-d99dd3390098-trusted-ca\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.610857 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.610839 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cef65b43-4a0c-4b34-a4e9-579f817084e1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qgqzr\" (UID: \"cef65b43-4a0c-4b34-a4e9-579f817084e1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:53.709905 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.709853 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5gs\" (UniqueName: \"kubernetes.io/projected/3ffa894e-03f3-4791-9afb-76d99ec77a24-kube-api-access-8l5gs\") pod \"downloads-6dd789c8b7-5vs22\" (UID: \"3ffa894e-03f3-4791-9afb-76d99ec77a24\") " pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.709905 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.709888 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e412acfd-c25b-4802-8d58-d99dd3390098-installation-pull-secrets\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.709909 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e412acfd-c25b-4802-8d58-d99dd3390098-image-registry-private-configuration\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.709932 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.709955 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-bound-sa-token\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.709984 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5vm\" (UniqueName: \"kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710036 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710016 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710045 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ffa894e-03f3-4791-9afb-76d99ec77a24-tmp\") pod \"downloads-6dd789c8b7-5vs22\" (UID: \"3ffa894e-03f3-4791-9afb-76d99ec77a24\") " pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710074 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710103 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710130 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc86s\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-kube-api-access-zc86s\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710156 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-registry-tls\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710179 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e412acfd-c25b-4802-8d58-d99dd3390098-ca-trust-extracted\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710207 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e412acfd-c25b-4802-8d58-d99dd3390098-trusted-ca\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710247 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710303 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.710459 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710331 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e412acfd-c25b-4802-8d58-d99dd3390098-registry-certificates\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.711010 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.711010 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.710683 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ffa894e-03f3-4791-9afb-76d99ec77a24-tmp\") pod \"downloads-6dd789c8b7-5vs22\" (UID: \"3ffa894e-03f3-4791-9afb-76d99ec77a24\") " pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.711106 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.711021 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.711106 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.711044 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.711311 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.711287 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e412acfd-c25b-4802-8d58-d99dd3390098-ca-trust-extracted\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.711722 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.711675 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e412acfd-c25b-4802-8d58-d99dd3390098-trusted-ca\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.712082 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.711961 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.712304 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.712280 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e412acfd-c25b-4802-8d58-d99dd3390098-registry-certificates\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.713001 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.712979 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e412acfd-c25b-4802-8d58-d99dd3390098-image-registry-private-configuration\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.713099 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.713037 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e412acfd-c25b-4802-8d58-d99dd3390098-installation-pull-secrets\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.713839 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.713819 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.713839 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.713833 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.714228 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.714201 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-registry-tls\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.739272 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.739249 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5vm\" (UniqueName: \"kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm\") pod \"console-5c69847f79-f92bg\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.739429 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.739373 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-bound-sa-token\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.739500 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.739481 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5gs\" (UniqueName: \"kubernetes.io/projected/3ffa894e-03f3-4791-9afb-76d99ec77a24-kube-api-access-8l5gs\") pod \"downloads-6dd789c8b7-5vs22\" (UID: \"3ffa894e-03f3-4791-9afb-76d99ec77a24\") " pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.740002 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.739986 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc86s\" (UniqueName: \"kubernetes.io/projected/e412acfd-c25b-4802-8d58-d99dd3390098-kube-api-access-zc86s\") pod \"image-registry-6dd666768c-vdtlt\" (UID: \"e412acfd-c25b-4802-8d58-d99dd3390098\") " pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:53.763200 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.763177 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:53.777387 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.777366 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:58:53.779980 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.779959 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:58:53.848944 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:53.848918 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:54.047272 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.047196 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr"] Oct 30 23:58:54.050117 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:54.050085 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef65b43_4a0c_4b34_a4e9_579f817084e1.slice/crio-af8eb533093bc2af6a2dd9adf8034bec9285eca902e8cbe557f4c29971ecc51d WatchSource:0}: Error finding container af8eb533093bc2af6a2dd9adf8034bec9285eca902e8cbe557f4c29971ecc51d: Status 404 returned error can't find the container with id af8eb533093bc2af6a2dd9adf8034bec9285eca902e8cbe557f4c29971ecc51d Oct 30 23:58:54.063137 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.063113 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5vs22"] Oct 30 23:58:54.066054 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:54.066027 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffa894e_03f3_4791_9afb_76d99ec77a24.slice/crio-0a3c86016b9ae425c2d946470360db9fefa290f6feb63148b3a54916c91a5c7d WatchSource:0}: Error finding container 0a3c86016b9ae425c2d946470360db9fefa290f6feb63148b3a54916c91a5c7d: Status 404 returned error can't find the container with id 0a3c86016b9ae425c2d946470360db9fefa290f6feb63148b3a54916c91a5c7d Oct 30 23:58:54.078142 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.078121 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c69847f79-f92bg"] Oct 30 23:58:54.080531 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:54.080506 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b9f51a_19c3_40f9_a100_d1fc7d19649a.slice/crio-ae0366e15860ee6b098426cf1ee86af6c8a1cd1522bec6f985620e3a8c686b22 WatchSource:0}: Error finding container ae0366e15860ee6b098426cf1ee86af6c8a1cd1522bec6f985620e3a8c686b22: Status 404 returned error can't find the container with id ae0366e15860ee6b098426cf1ee86af6c8a1cd1522bec6f985620e3a8c686b22 Oct 30 23:58:54.137204 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.137180 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd666768c-vdtlt"] Oct 30 23:58:54.139946 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:54.139926 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode412acfd_c25b_4802_8d58_d99dd3390098.slice/crio-f141de6e6e68af110385f251965c8d5019697182a085f1c8262fec36975b7bb9 WatchSource:0}: Error finding container f141de6e6e68af110385f251965c8d5019697182a085f1c8262fec36975b7bb9: Status 404 returned error can't find the container with id f141de6e6e68af110385f251965c8d5019697182a085f1c8262fec36975b7bb9 Oct 30 23:58:54.236615 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.236581 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" event={"ID":"cef65b43-4a0c-4b34-a4e9-579f817084e1","Type":"ContainerStarted","Data":"af8eb533093bc2af6a2dd9adf8034bec9285eca902e8cbe557f4c29971ecc51d"} Oct 30 23:58:54.239552 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.239497 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c69847f79-f92bg" event={"ID":"14b9f51a-19c3-40f9-a100-d1fc7d19649a","Type":"ContainerStarted","Data":"ae0366e15860ee6b098426cf1ee86af6c8a1cd1522bec6f985620e3a8c686b22"} Oct 30 23:58:54.241559 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.241529 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" event={"ID":"e412acfd-c25b-4802-8d58-d99dd3390098","Type":"ContainerStarted","Data":"7c18470e49863812cede8d33faf20ac3040cdb73b57651ec73385ebfd4e08b81"} Oct 30 23:58:54.241658 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.241567 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" event={"ID":"e412acfd-c25b-4802-8d58-d99dd3390098","Type":"ContainerStarted","Data":"f141de6e6e68af110385f251965c8d5019697182a085f1c8262fec36975b7bb9"} Oct 30 23:58:54.241658 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.241615 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:58:54.242961 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.242942 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5vs22" event={"ID":"3ffa894e-03f3-4791-9afb-76d99ec77a24","Type":"ContainerStarted","Data":"0a3c86016b9ae425c2d946470360db9fefa290f6feb63148b3a54916c91a5c7d"} Oct 30 23:58:54.265736 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:54.265678 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" podStartSLOduration=1.265660592 podStartE2EDuration="1.265660592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:54.264236697 +0000 UTC m=+18.325011391" watchObservedRunningTime="2026-10-30 23:58:54.265660592 +0000 UTC m=+18.326435288" Oct 30 23:58:56.259502 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:56.259461 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" event={"ID":"cef65b43-4a0c-4b34-a4e9-579f817084e1","Type":"ContainerStarted","Data":"6dfce9dc3f7a777534a5a2806f5e982c97748d10c42dcc99f403d1c303114346"} Oct 30 23:58:56.259992 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:56.259838 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:56.265800 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:56.265753 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" Oct 30 23:58:56.278445 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:56.278389 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qgqzr" podStartSLOduration=1.848240441 podStartE2EDuration="3.278377513s" totalImagesPullingTime="1.430137072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:56.27723903 +0000 UTC m=+20.338013726" watchObservedRunningTime="2026-10-30 23:58:56.278377513 +0000 UTC m=+20.339152205" Oct 30 23:58:57.319777 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.318103 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b"] Oct 30 23:58:57.327299 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.327268 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.334372 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.334333 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b"] Oct 30 23:58:57.335665 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.335124 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 23:58:57.335665 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.335189 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 23:58:57.335665 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.335655 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c8jrw\"" Oct 30 23:58:57.336493 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.335815 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:58:57.456291 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.456249 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9ps\" (UniqueName: \"kubernetes.io/projected/e1d53cc2-f01d-4b48-91b3-23717c206abd-kube-api-access-km9ps\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.456438 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.456305 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1d53cc2-f01d-4b48-91b3-23717c206abd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.456438 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.456363 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1d53cc2-f01d-4b48-91b3-23717c206abd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.456438 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.456379 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1d53cc2-f01d-4b48-91b3-23717c206abd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.557763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.557284 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1d53cc2-f01d-4b48-91b3-23717c206abd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.557763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.557320 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1d53cc2-f01d-4b48-91b3-23717c206abd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.557763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.557371 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9ps\" (UniqueName: \"kubernetes.io/projected/e1d53cc2-f01d-4b48-91b3-23717c206abd-kube-api-access-km9ps\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.557763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.557413 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1d53cc2-f01d-4b48-91b3-23717c206abd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.558276 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.558156 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1d53cc2-f01d-4b48-91b3-23717c206abd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.560571 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.560497 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1d53cc2-f01d-4b48-91b3-23717c206abd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.561464 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.561434 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1d53cc2-f01d-4b48-91b3-23717c206abd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.562096 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.562076 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-98c775b85-mpvpw"] Oct 30 23:58:57.567252 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.567231 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.573916 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.573867 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9ps\" (UniqueName: \"kubernetes.io/projected/e1d53cc2-f01d-4b48-91b3-23717c206abd-kube-api-access-km9ps\") pod \"prometheus-operator-9cd89b74c-2ng7b\" (UID: \"e1d53cc2-f01d-4b48-91b3-23717c206abd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.578079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.577170 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98c775b85-mpvpw"] Oct 30 23:58:57.579461 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.579435 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 23:58:57.645766 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.645737 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" Oct 30 23:58:57.759663 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.759635 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.759830 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.759678 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.759830 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.759733 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.759980 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.759767 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.760171 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.760154 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpv2l\" (UniqueName: \"kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.760391 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.760376 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.760570 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.760555 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.760724 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.760596 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.861919 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.861846 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.861919 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.861885 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv2l\" (UniqueName: \"kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.861919 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.861914 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.861943 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.861966 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.862045 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.862073 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.862135 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862625 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.862595 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862769 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.862749 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.862936 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.862915 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.863316 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.863250 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.863816 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.863757 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.865292 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.865271 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.865571 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.865548 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.875674 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.875653 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpv2l\" (UniqueName: \"kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l\") pod \"console-98c775b85-mpvpw\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.902489 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.902469 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:58:57.941158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:57.941101 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b"] Oct 30 23:58:57.941990 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:57.941962 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d53cc2_f01d_4b48_91b3_23717c206abd.slice/crio-46fc3a87341f0051badc046f5a8d731e59f6e01e905f5ec846c4bc031081e504 WatchSource:0}: Error finding container 46fc3a87341f0051badc046f5a8d731e59f6e01e905f5ec846c4bc031081e504: Status 404 returned error can't find the container with id 46fc3a87341f0051badc046f5a8d731e59f6e01e905f5ec846c4bc031081e504 Oct 30 23:58:58.188429 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.188334 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98c775b85-mpvpw"] Oct 30 23:58:58.192523 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:58:58.192496 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d12cf1_c32e_4e32_8df3_8221274ee956.slice/crio-80b0027270fcb8a3033d520d9b79b24f68f0a38ab4b603e38dc009c2ae42a18d WatchSource:0}: Error finding container 80b0027270fcb8a3033d520d9b79b24f68f0a38ab4b603e38dc009c2ae42a18d: Status 404 returned error can't find the container with id 80b0027270fcb8a3033d520d9b79b24f68f0a38ab4b603e38dc009c2ae42a18d Oct 30 23:58:58.272309 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.272273 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98c775b85-mpvpw" event={"ID":"29d12cf1-c32e-4e32-8df3-8221274ee956","Type":"ContainerStarted","Data":"203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21"} Oct 30 23:58:58.272462 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.272340 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98c775b85-mpvpw" event={"ID":"29d12cf1-c32e-4e32-8df3-8221274ee956","Type":"ContainerStarted","Data":"80b0027270fcb8a3033d520d9b79b24f68f0a38ab4b603e38dc009c2ae42a18d"} Oct 30 23:58:58.274371 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.274322 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c69847f79-f92bg" event={"ID":"14b9f51a-19c3-40f9-a100-d1fc7d19649a","Type":"ContainerStarted","Data":"1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1"} Oct 30 23:58:58.275743 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.275716 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" event={"ID":"e1d53cc2-f01d-4b48-91b3-23717c206abd","Type":"ContainerStarted","Data":"46fc3a87341f0051badc046f5a8d731e59f6e01e905f5ec846c4bc031081e504"} Oct 30 23:58:58.298643 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.298597 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-98c775b85-mpvpw" podStartSLOduration=1.298580182 podStartE2EDuration="1.298580182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:58.297566962 +0000 UTC m=+22.358341656" watchObservedRunningTime="2026-10-30 23:58:58.298580182 +0000 UTC m=+22.359354879" Oct 30 23:58:58.321086 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:58:58.321038 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c69847f79-f92bg" podStartSLOduration=1.7379457999999999 podStartE2EDuration="5.321024239s" totalImagesPullingTime="3.583078439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:58.31918959 +0000 UTC m=+22.379964285" watchObservedRunningTime="2026-10-30 23:58:58.321024239 +0000 UTC m=+22.381798934" Oct 30 23:59:00.159919 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:00.159846 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:59:00.287242 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:00.287206 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" event={"ID":"e1d53cc2-f01d-4b48-91b3-23717c206abd","Type":"ContainerStarted","Data":"14f3512524f01fafe18623a14abb793b0390ef40f73b26c22bd8bfed6049764a"} Oct 30 23:59:00.287423 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:00.287267 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" event={"ID":"e1d53cc2-f01d-4b48-91b3-23717c206abd","Type":"ContainerStarted","Data":"df6922ae230409eafd04644ce59d12092eeea78d538ac5763c71b77fe84ffa47"} Oct 30 23:59:00.308532 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:00.308477 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2ng7b" podStartSLOduration=1.473803324 podStartE2EDuration="3.308464174s" totalImagesPullingTime="1.83466085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:00.307991731 +0000 UTC m=+24.368766427" watchObservedRunningTime="2026-10-30 23:59:00.308464174 +0000 UTC m=+24.369238869" Oct 30 23:59:02.801165 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.801132 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nh72c"] Oct 30 23:59:02.827120 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.827097 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.840367 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.840340 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xcdsb\"" Oct 30 23:59:02.840950 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.840930 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:59:02.841060 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.841020 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:59:02.871178 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.871152 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:59:02.906665 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906637 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-tls\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906670 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-accelerators-collector-config\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906692 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-wtmp\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906846 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906796 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-textfile\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906887 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906855 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-sys\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906887 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906875 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08c22c12-34b8-4338-9558-81ad92772a96-metrics-client-ca\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906948 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906891 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xj5\" (UniqueName: \"kubernetes.io/projected/08c22c12-34b8-4338-9558-81ad92772a96-kube-api-access-t8xj5\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.906987 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906973 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:02.907022 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:02.906995 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-root\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.007957 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.007933 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-tls\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.007962 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-accelerators-collector-config\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.007985 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-wtmp\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008005 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-textfile\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008036 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-sys\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008063 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08c22c12-34b8-4338-9558-81ad92772a96-metrics-client-ca\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008329 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008086 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xj5\" (UniqueName: \"kubernetes.io/projected/08c22c12-34b8-4338-9558-81ad92772a96-kube-api-access-t8xj5\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008329 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008138 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-sys\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008329 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008140 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008329 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008160 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-wtmp\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008329 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008268 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-root\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008580 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008375 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08c22c12-34b8-4338-9558-81ad92772a96-root\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008580 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008455 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-textfile\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008688 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008611 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-accelerators-collector-config\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.008688 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.008640 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08c22c12-34b8-4338-9558-81ad92772a96-metrics-client-ca\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.010628 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.010606 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-tls\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.010739 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.010686 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08c22c12-34b8-4338-9558-81ad92772a96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.020717 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.020697 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xj5\" (UniqueName: \"kubernetes.io/projected/08c22c12-34b8-4338-9558-81ad92772a96-kube-api-access-t8xj5\") pod \"node-exporter-nh72c\" (UID: \"08c22c12-34b8-4338-9558-81ad92772a96\") " pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.045374 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.045356 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6gwvc" Oct 30 23:59:03.200184 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.200113 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nh72c" Oct 30 23:59:03.249925 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:59:03.249888 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c22c12_34b8_4338_9558_81ad92772a96.slice/crio-1a844de34f433a2e34c47d42577a9db72d12b9c48ce1945c239db1b13478a1eb WatchSource:0}: Error finding container 1a844de34f433a2e34c47d42577a9db72d12b9c48ce1945c239db1b13478a1eb: Status 404 returned error can't find the container with id 1a844de34f433a2e34c47d42577a9db72d12b9c48ce1945c239db1b13478a1eb Oct 30 23:59:03.305182 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.305142 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nh72c" event={"ID":"08c22c12-34b8-4338-9558-81ad92772a96","Type":"ContainerStarted","Data":"1a844de34f433a2e34c47d42577a9db72d12b9c48ce1945c239db1b13478a1eb"} Oct 30 23:59:03.473099 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.473026 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:59:03.781150 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.781099 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:59:03.781320 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.781167 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:59:03.786210 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:03.786173 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:59:04.317008 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:04.316966 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:59:05.317985 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:05.317947 5073 generic.go:356] "Generic (PLEG): container finished" podID="08c22c12-34b8-4338-9558-81ad92772a96" containerID="4a1c7deca3a17caa02d8a5ccd47cebae1ba776e7cb79e79cb934b544eee93982" exitCode=0 Oct 30 23:59:05.318514 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:05.318053 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nh72c" event={"ID":"08c22c12-34b8-4338-9558-81ad92772a96","Type":"ContainerDied","Data":"4a1c7deca3a17caa02d8a5ccd47cebae1ba776e7cb79e79cb934b544eee93982"} Oct 30 23:59:06.324488 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:06.324447 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nh72c" event={"ID":"08c22c12-34b8-4338-9558-81ad92772a96","Type":"ContainerStarted","Data":"01318b80a4e94f8e18394e718ec9f5e21623148d302f981e70b4c7e85d943b52"} Oct 30 23:59:06.324871 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:06.324519 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nh72c" event={"ID":"08c22c12-34b8-4338-9558-81ad92772a96","Type":"ContainerStarted","Data":"10f114df2188c40b9df4402c9aa8dd131e9b8a74ba48453408a05cffe1ae72b8"} Oct 30 23:59:06.349049 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:06.349007 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nh72c" podStartSLOduration=2.696049232 podStartE2EDuration="4.34899465s" totalImagesPullingTime="1.555578856s" totalInitContainerRuntime="97.366562ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:06.347159213 +0000 UTC m=+30.407933899" watchObservedRunningTime="2026-10-30 23:59:06.34899465 +0000 UTC m=+30.409769360" Oct 30 23:59:07.903520 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:07.902729 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:59:07.903520 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:07.902836 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:59:07.909236 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:07.909207 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:59:08.341072 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:08.341047 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:59:08.404511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:08.403946 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c69847f79-f92bg"] Oct 30 23:59:10.193765 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.193731 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65645f9684-k9zmb"] Oct 30 23:59:10.227597 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.227569 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65645f9684-k9zmb"] Oct 30 23:59:10.227767 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.227677 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280514 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280485 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280645 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280531 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280645 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280550 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280645 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280578 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280805 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280680 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280805 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280740 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280805 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280765 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.280929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.280824 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4lg\" (UniqueName: \"kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.381840 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.381807 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4lg\" (UniqueName: \"kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382025 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.381883 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382025 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.381928 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382025 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.381951 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382370 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382107 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382370 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382160 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382370 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382210 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382370 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382247 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382659 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382635 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382747 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382707 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.382816 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.382741 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.383020 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.383001 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.383155 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.383131 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.385241 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.385216 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.391104 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.391085 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.394617 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.394593 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4lg\" (UniqueName: \"kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg\") pod \"console-65645f9684-k9zmb\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.550097 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.550062 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:10.858799 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:10.858696 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65645f9684-k9zmb"] Oct 30 23:59:10.860865 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:59:10.860836 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21fb55f_23a4_4c77_a4fc_79d9de4ee70f.slice/crio-142880b9a10dbb32993c2c325a1ee946a91003cff5eafc9477a9364aede7a2d9 WatchSource:0}: Error finding container 142880b9a10dbb32993c2c325a1ee946a91003cff5eafc9477a9364aede7a2d9: Status 404 returned error can't find the container with id 142880b9a10dbb32993c2c325a1ee946a91003cff5eafc9477a9364aede7a2d9 Oct 30 23:59:11.356220 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:11.356180 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65645f9684-k9zmb" event={"ID":"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f","Type":"ContainerStarted","Data":"040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c"} Oct 30 23:59:11.356716 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:11.356243 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65645f9684-k9zmb" event={"ID":"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f","Type":"ContainerStarted","Data":"142880b9a10dbb32993c2c325a1ee946a91003cff5eafc9477a9364aede7a2d9"} Oct 30 23:59:11.378929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:11.378876 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65645f9684-k9zmb" podStartSLOduration=1.378856872 podStartE2EDuration="1.378856872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:11.37846025 +0000 UTC m=+35.439234945" watchObservedRunningTime="2026-10-30 23:59:11.378856872 +0000 UTC m=+35.439631567" Oct 30 23:59:13.088375 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:59:13.088331 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 30 23:59:14.374009 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:14.373955 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5vs22" event={"ID":"3ffa894e-03f3-4791-9afb-76d99ec77a24","Type":"ContainerStarted","Data":"f6cc8ca20a6190900e13794ef90e3dc3a0ec52106f0b16b008f8e7e785d1c78c"} Oct 30 23:59:14.374561 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:14.374349 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:59:14.388186 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:14.388159 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5vs22" Oct 30 23:59:14.397248 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:14.397161 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-5vs22" podStartSLOduration=1.724659033 podStartE2EDuration="21.397144162s" totalImagesPullingTime="19.672485129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:14.395329689 +0000 UTC m=+38.456104384" watchObservedRunningTime="2026-10-30 23:59:14.397144162 +0000 UTC m=+38.457918870" Oct 30 23:59:15.254896 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:15.254866 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd666768c-vdtlt" Oct 30 23:59:15.312252 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:15.312215 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59457478cb-rqvwg"] Oct 30 23:59:16.783420 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.783388 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65645f9684-k9zmb"] Oct 30 23:59:16.827635 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.827608 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-777cbb6f9c-tzl4n"] Oct 30 23:59:16.859163 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.859136 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777cbb6f9c-tzl4n"] Oct 30 23:59:16.859331 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.859256 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952199 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952163 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952357 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952221 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952357 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952290 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952357 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952349 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdl2\" (UniqueName: \"kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952531 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952510 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952577 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952547 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952667 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952645 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:16.952796 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:16.952710 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.053818 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.053742 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.053818 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.053800 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054022 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.053819 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054022 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.053843 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdl2\" (UniqueName: \"kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054022 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.053931 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054022 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.053952 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054022 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.054007 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054332 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.054044 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.054423 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.054401 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.055045 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.055016 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.055156 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.055136 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.055333 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.055303 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.055511 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.055486 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.056914 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.056895 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.057385 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.057217 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.067517 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.067491 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdl2\" (UniqueName: \"kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2\") pod \"console-777cbb6f9c-tzl4n\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.193048 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.193014 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:17.509424 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:17.509387 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777cbb6f9c-tzl4n"] Oct 30 23:59:17.515227 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:59:17.515194 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9844dc0_181b_4b78_8fcb_c50586b024b9.slice/crio-2ba186af7a761e98e05ccaac742738db47a4205381eb7f4c3f3ea34295a2ff47 WatchSource:0}: Error finding container 2ba186af7a761e98e05ccaac742738db47a4205381eb7f4c3f3ea34295a2ff47: Status 404 returned error can't find the container with id 2ba186af7a761e98e05ccaac742738db47a4205381eb7f4c3f3ea34295a2ff47 Oct 30 23:59:18.399473 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.399427 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777cbb6f9c-tzl4n" event={"ID":"b9844dc0-181b-4b78-8fcb-c50586b024b9","Type":"ContainerStarted","Data":"be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4"} Oct 30 23:59:18.399998 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.399496 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777cbb6f9c-tzl4n" event={"ID":"b9844dc0-181b-4b78-8fcb-c50586b024b9","Type":"ContainerStarted","Data":"2ba186af7a761e98e05ccaac742738db47a4205381eb7f4c3f3ea34295a2ff47"} Oct 30 23:59:18.423742 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.423687 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-777cbb6f9c-tzl4n" podStartSLOduration=2.423672311 podStartE2EDuration="2.423672311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:18.420482997 +0000 UTC m=+42.481257694" watchObservedRunningTime="2026-10-30 23:59:18.423672311 +0000 UTC m=+42.484447002" Oct 30 23:59:18.492672 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.492611 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" containerID="cri-o://ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f" gracePeriod=30 Oct 30 23:59:18.808606 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.808574 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:59:18.873210 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873179 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873384 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873245 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873472 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873445 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873529 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873493 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873582 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873525 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873639 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873579 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873639 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873625 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.873745 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873652 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:18.873745 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.873705 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") pod \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\" (UID: \"500001f6-9b33-4347-b5f5-cfcf79e6ed09\") " Oct 30 23:59:18.874056 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.874032 5073 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-certificates\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.874740 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.874711 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:18.880220 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.879262 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "kube-api-access-jkrxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.880220 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.879347 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.880220 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.879552 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:18.880982 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.880864 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:18.881157 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.881121 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:18.882422 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.882398 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted" pod "500001f6-9b33-4347-b5f5-cfcf79e6ed09" (UID: "500001f6-9b33-4347-b5f5-cfcf79e6ed09"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:18.975143 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975078 5073 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-image-registry-private-configuration\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.975143 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975104 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrxj\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-kube-api-access-jkrxj\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.975143 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975115 5073 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-registry-tls\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.975143 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975124 5073 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/500001f6-9b33-4347-b5f5-cfcf79e6ed09-bound-sa-token\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.975143 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975136 5073 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/500001f6-9b33-4347-b5f5-cfcf79e6ed09-installation-pull-secrets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.975453 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975149 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/500001f6-9b33-4347-b5f5-cfcf79e6ed09-trusted-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:18.975453 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:18.975158 5073 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/500001f6-9b33-4347-b5f5-cfcf79e6ed09-ca-trust-extracted\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:19.406744 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.406704 5073 generic.go:356] "Generic (PLEG): container finished" podID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerID="ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f" exitCode=0 Oct 30 23:59:19.407226 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.406759 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" event={"ID":"500001f6-9b33-4347-b5f5-cfcf79e6ed09","Type":"ContainerDied","Data":"ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f"} Oct 30 23:59:19.407226 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.406809 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" Oct 30 23:59:19.407226 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.406839 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69cf5d76f-vwpgd" event={"ID":"500001f6-9b33-4347-b5f5-cfcf79e6ed09","Type":"ContainerDied","Data":"eec32dd11618bffa442f1a3f5f515264e118b77cfb0c850e383dc57249230d1c"} Oct 30 23:59:19.407226 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.406866 5073 scope.go:122] "RemoveContainer" containerID="ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f" Oct 30 23:59:19.417820 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.417775 5073 scope.go:122] "RemoveContainer" containerID="ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f" Oct 30 23:59:19.418164 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:59:19.418139 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f\": container with ID starting with ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f not found: ID does not exist" containerID="ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f" Oct 30 23:59:19.418269 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.418172 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f"} err="failed to get container status \"ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f\": rpc error: code = NotFound desc = could not find container \"ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f\": container with ID starting with ef217db64e1f8dc6b8951d58a984d8636945dcefd8574cd297004545803cec5f not found: ID does not exist" Oct 30 23:59:19.436997 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.436970 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69cf5d76f-vwpgd"] Oct 30 23:59:19.442952 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:19.442923 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69cf5d76f-vwpgd"] Oct 30 23:59:20.060094 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:20.060058 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" path="/var/lib/kubelet/pods/500001f6-9b33-4347-b5f5-cfcf79e6ed09/volumes" Oct 30 23:59:20.550304 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:20.550269 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:27.194091 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:27.194059 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:27.194613 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:27.194120 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:27.198950 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:27.198927 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:27.449150 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:27.449080 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 30 23:59:27.500201 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:27.500159 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-98c775b85-mpvpw"] Oct 30 23:59:33.427527 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.427468 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c69847f79-f92bg" podUID="14b9f51a-19c3-40f9-a100-d1fc7d19649a" containerName="console" containerID="cri-o://1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1" gracePeriod=15 Oct 30 23:59:33.700505 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.700447 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:59:33.803839 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.803813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.803976 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.803862 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.803976 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.803889 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.804080 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804006 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm\" (UniqueName: \"kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.804080 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804069 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.804185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804099 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.804185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804099 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:33.804185 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804126 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config\") pod \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\" (UID: \"14b9f51a-19c3-40f9-a100-d1fc7d19649a\") " Oct 30 23:59:33.804338 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804255 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:33.804485 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804456 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:33.804570 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804463 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:33.804614 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804583 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b9f51a-19c3-40f9-a100-d1fc7d19649a-tmp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:33.804614 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804605 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-service-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:33.804694 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804620 5073 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:33.804694 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.804638 5073 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b9f51a-19c3-40f9-a100-d1fc7d19649a-oauth-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:33.806309 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.806285 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:33.806426 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.806326 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:33.806426 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.806392 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm" pod "14b9f51a-19c3-40f9-a100-d1fc7d19649a" (UID: "14b9f51a-19c3-40f9-a100-d1fc7d19649a"). InnerVolumeSpecName "kube-api-access-8b5vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:33.905715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.905692 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b5vm\" (UniqueName: \"kubernetes.io/projected/14b9f51a-19c3-40f9-a100-d1fc7d19649a-kube-api-access-8b5vm\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:33.905715 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.905712 5073 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-oauth-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:33.905896 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:33.905725 5073 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b9f51a-19c3-40f9-a100-d1fc7d19649a-console-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:34.475723 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.475647 5073 generic.go:356] "Generic (PLEG): container finished" podID="14b9f51a-19c3-40f9-a100-d1fc7d19649a" containerID="1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1" exitCode=0 Oct 30 23:59:34.475723 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.475698 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c69847f79-f92bg" Oct 30 23:59:34.476197 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.475695 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c69847f79-f92bg" event={"ID":"14b9f51a-19c3-40f9-a100-d1fc7d19649a","Type":"ContainerDied","Data":"1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1"} Oct 30 23:59:34.476197 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.475840 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c69847f79-f92bg" event={"ID":"14b9f51a-19c3-40f9-a100-d1fc7d19649a","Type":"ContainerDied","Data":"ae0366e15860ee6b098426cf1ee86af6c8a1cd1522bec6f985620e3a8c686b22"} Oct 30 23:59:34.476197 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.475871 5073 scope.go:122] "RemoveContainer" containerID="1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1" Oct 30 23:59:34.484531 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.484501 5073 scope.go:122] "RemoveContainer" containerID="1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1" Oct 30 23:59:34.484830 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:59:34.484808 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1\": container with ID starting with 1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1 not found: ID does not exist" containerID="1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1" Oct 30 23:59:34.484883 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.484847 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1"} err="failed to get container status \"1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1\": rpc error: code = NotFound desc = could not find container \"1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1\": container with ID starting with 1bce28bf180cabb082c123a2958d41ebe7b508a1de1da4c1cbb30849abd49dd1 not found: ID does not exist" Oct 30 23:59:34.495430 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.495410 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c69847f79-f92bg"] Oct 30 23:59:34.500008 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:34.499989 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c69847f79-f92bg"] Oct 30 23:59:36.058562 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:36.058532 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b9f51a-19c3-40f9-a100-d1fc7d19649a" path="/var/lib/kubelet/pods/14b9f51a-19c3-40f9-a100-d1fc7d19649a/volumes" Oct 30 23:59:40.342607 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.342552 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" podUID="10737b2e-c98b-433d-a846-c087bfc5410f" containerName="registry" containerID="cri-o://009cdd1c97200f312e2a61a43db65c160d0b1b360ed3072edc223d182187c6e1" gracePeriod=30 Oct 30 23:59:40.508125 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.508092 5073 generic.go:356] "Generic (PLEG): container finished" podID="10737b2e-c98b-433d-a846-c087bfc5410f" containerID="009cdd1c97200f312e2a61a43db65c160d0b1b360ed3072edc223d182187c6e1" exitCode=0 Oct 30 23:59:40.508275 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.508162 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" event={"ID":"10737b2e-c98b-433d-a846-c087bfc5410f","Type":"ContainerDied","Data":"009cdd1c97200f312e2a61a43db65c160d0b1b360ed3072edc223d182187c6e1"} Oct 30 23:59:40.677078 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.677043 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:59:40.868083 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868054 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868264 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868094 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868264 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868149 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868264 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868171 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868264 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868200 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868264 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868234 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868264 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868258 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868574 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868303 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") pod \"10737b2e-c98b-433d-a846-c087bfc5410f\" (UID: \"10737b2e-c98b-433d-a846-c087bfc5410f\") " Oct 30 23:59:40.868671 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868522 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:40.868911 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.868884 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:40.871122 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.871081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:40.871386 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.871359 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:40.871498 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.871382 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:40.871540 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.871487 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:40.871540 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.871489 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "kube-api-access-gmwjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:40.876917 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.876890 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted" pod "10737b2e-c98b-433d-a846-c087bfc5410f" (UID: "10737b2e-c98b-433d-a846-c087bfc5410f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:40.969340 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969275 5073 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-image-registry-private-configuration\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969340 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969302 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmwjt\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-kube-api-access-gmwjt\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969340 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969318 5073 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10737b2e-c98b-433d-a846-c087bfc5410f-installation-pull-secrets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969340 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969331 5073 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10737b2e-c98b-433d-a846-c087bfc5410f-ca-trust-extracted\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969537 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969346 5073 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-bound-sa-token\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969537 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969359 5073 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-registry-certificates\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969537 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969373 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10737b2e-c98b-433d-a846-c087bfc5410f-trusted-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:40.969537 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:40.969386 5073 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10737b2e-c98b-433d-a846-c087bfc5410f-registry-tls\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:41.513407 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:41.513381 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" Oct 30 23:59:41.513407 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:41.513399 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59457478cb-rqvwg" event={"ID":"10737b2e-c98b-433d-a846-c087bfc5410f","Type":"ContainerDied","Data":"94d4770c132787315ff6184d87286c2648a4a9d088a524a0161a4d7d0752e6e8"} Oct 30 23:59:41.513846 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:41.513451 5073 scope.go:122] "RemoveContainer" containerID="009cdd1c97200f312e2a61a43db65c160d0b1b360ed3072edc223d182187c6e1" Oct 30 23:59:41.536697 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:41.536674 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59457478cb-rqvwg"] Oct 30 23:59:41.542155 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:41.542135 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59457478cb-rqvwg"] Oct 30 23:59:41.806541 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:41.806451 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-65645f9684-k9zmb" podUID="c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" containerName="console" containerID="cri-o://040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c" gracePeriod=15 Oct 30 23:59:42.058108 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.058082 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10737b2e-c98b-433d-a846-c087bfc5410f" path="/var/lib/kubelet/pods/10737b2e-c98b-433d-a846-c087bfc5410f/volumes" Oct 30 23:59:42.095256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.095231 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:42.224335 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224301 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bdd949fc8-ct86f"] Oct 30 23:59:42.224814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224770 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" containerName="console" Oct 30 23:59:42.224814 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224811 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" containerName="console" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224828 5073 cpu_manager.go:414] "containerMap: removing container" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224833 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224847 5073 cpu_manager.go:414] "containerMap: removing container" podUID="10737b2e-c98b-433d-a846-c087bfc5410f" containerName="registry" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224852 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10737b2e-c98b-433d-a846-c087bfc5410f" containerName="registry" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224869 5073 cpu_manager.go:414] "containerMap: removing container" podUID="14b9f51a-19c3-40f9-a100-d1fc7d19649a" containerName="console" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224873 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b9f51a-19c3-40f9-a100-d1fc7d19649a" containerName="console" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224938 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="500001f6-9b33-4347-b5f5-cfcf79e6ed09" containerName="registry" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224947 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="10737b2e-c98b-433d-a846-c087bfc5410f" containerName="registry" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224955 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b9f51a-19c3-40f9-a100-d1fc7d19649a" containerName="console" Oct 30 23:59:42.224996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.224961 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" containerName="console" Oct 30 23:59:42.243048 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.243023 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdd949fc8-ct86f"] Oct 30 23:59:42.243184 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.243158 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.279046 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279015 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279046 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279048 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279235 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279078 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279235 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279100 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg\" (UniqueName: \"kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279235 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279138 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279235 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279203 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279235 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279222 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279477 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279250 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert\") pod \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\" (UID: \"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f\") " Oct 30 23:59:42.279477 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279278 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:42.279585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279485 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:42.279585 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279493 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-tmp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.279694 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279595 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:42.279743 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279694 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:42.279743 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.279727 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:42.281414 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.281385 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "kube-api-access-8s4lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:42.281529 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.281508 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:42.281577 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.281524 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config" pod "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" (UID: "c21fb55f-23a4-4c77-a4fc-79d9de4ee70f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:42.380108 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380052 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-tmp\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380108 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380085 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-service-ca\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380241 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380109 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-oauth-config\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380241 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380131 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-config\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380303 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380260 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-oauth-serving-cert\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380303 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380291 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fnn\" (UniqueName: \"kubernetes.io/projected/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-kube-api-access-r2fnn\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380379 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380360 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-trusted-ca-bundle\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380411 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380390 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-serving-cert\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.380448 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380438 5073 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-oauth-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.380488 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380474 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4lg\" (UniqueName: \"kubernetes.io/projected/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-kube-api-access-8s4lg\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.380488 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380486 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-trusted-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.380550 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380496 5073 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.380550 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380505 5073 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-console-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.380550 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380513 5073 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-oauth-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.380550 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.380523 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f-service-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:42.481322 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481299 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-tmp\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481327 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-service-ca\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481352 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-oauth-config\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481369 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-config\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481402 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-oauth-serving-cert\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481433 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481425 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fnn\" (UniqueName: \"kubernetes.io/projected/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-kube-api-access-r2fnn\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481455 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-trusted-ca-bundle\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481478 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-serving-cert\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.481763 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.481721 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-tmp\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.482097 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.482075 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-service-ca\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.482261 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.482239 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-config\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.482313 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.482259 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-oauth-serving-cert\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.482530 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.482510 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-trusted-ca-bundle\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.483944 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.483924 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-oauth-config\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.484079 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.484060 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-console-serving-cert\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.493909 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.493892 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fnn\" (UniqueName: \"kubernetes.io/projected/0dd4a18f-ba45-441f-9f8a-e2fccc32b77c-kube-api-access-r2fnn\") pod \"console-7bdd949fc8-ct86f\" (UID: \"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c\") " pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.518617 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.518590 5073 generic.go:356] "Generic (PLEG): container finished" podID="c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" containerID="040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c" exitCode=0 Oct 30 23:59:42.518996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.518643 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65645f9684-k9zmb" Oct 30 23:59:42.518996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.518669 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65645f9684-k9zmb" event={"ID":"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f","Type":"ContainerDied","Data":"040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c"} Oct 30 23:59:42.518996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.518717 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65645f9684-k9zmb" event={"ID":"c21fb55f-23a4-4c77-a4fc-79d9de4ee70f","Type":"ContainerDied","Data":"142880b9a10dbb32993c2c325a1ee946a91003cff5eafc9477a9364aede7a2d9"} Oct 30 23:59:42.518996 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.518733 5073 scope.go:122] "RemoveContainer" containerID="040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c" Oct 30 23:59:42.531479 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.531460 5073 scope.go:122] "RemoveContainer" containerID="040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c" Oct 30 23:59:42.531729 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:59:42.531709 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c\": container with ID starting with 040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c not found: ID does not exist" containerID="040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c" Oct 30 23:59:42.531774 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.531737 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c"} err="failed to get container status \"040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c\": rpc error: code = NotFound desc = could not find container \"040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c\": container with ID starting with 040b3f91aa12668938f7ed8ba471bd8c9f0dd00d8a78aa9c173cfb6240626a2c not found: ID does not exist" Oct 30 23:59:42.543361 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.543338 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65645f9684-k9zmb"] Oct 30 23:59:42.552896 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.552870 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65645f9684-k9zmb"] Oct 30 23:59:42.568917 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.568888 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:42.880752 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:42.880720 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdd949fc8-ct86f"] Oct 30 23:59:42.884493 ip-10-0-140-246 kubenswrapper[5073]: W1030 23:59:42.884465 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd4a18f_ba45_441f_9f8a_e2fccc32b77c.slice/crio-e447f6a3e649c7a1234e2c900ddcc835d5ae28dff76d3f5893046114966f7611 WatchSource:0}: Error finding container e447f6a3e649c7a1234e2c900ddcc835d5ae28dff76d3f5893046114966f7611: Status 404 returned error can't find the container with id e447f6a3e649c7a1234e2c900ddcc835d5ae28dff76d3f5893046114966f7611 Oct 30 23:59:43.085632 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:59:43.085598 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 30 23:59:43.526156 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:43.526116 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdd949fc8-ct86f" event={"ID":"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c","Type":"ContainerStarted","Data":"85c127f59f44e22e934453b64243e739f5ee7387f5be935baf3f8a3492b77959"} Oct 30 23:59:43.526600 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:43.526181 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdd949fc8-ct86f" event={"ID":"0dd4a18f-ba45-441f-9f8a-e2fccc32b77c","Type":"ContainerStarted","Data":"e447f6a3e649c7a1234e2c900ddcc835d5ae28dff76d3f5893046114966f7611"} Oct 30 23:59:43.548487 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:43.548444 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7bdd949fc8-ct86f" podStartSLOduration=1.548429917 podStartE2EDuration="1.548429917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:43.546634089 +0000 UTC m=+67.607408822" watchObservedRunningTime="2026-10-30 23:59:43.548429917 +0000 UTC m=+67.609204613" Oct 30 23:59:44.059076 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:44.059042 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21fb55f-23a4-4c77-a4fc-79d9de4ee70f" path="/var/lib/kubelet/pods/c21fb55f-23a4-4c77-a4fc-79d9de4ee70f/volumes" Oct 30 23:59:45.538232 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:45.538198 5073 generic.go:356] "Generic (PLEG): container finished" podID="a2587ab9-e842-47c6-87b5-fb0b1ab8b76e" containerID="2057f0b41c2e1145df4c43efe708098fed2cd2168d2d4beacbca4f80ab50fb0b" exitCode=0 Oct 30 23:59:45.538676 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:45.538279 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" event={"ID":"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e","Type":"ContainerDied","Data":"2057f0b41c2e1145df4c43efe708098fed2cd2168d2d4beacbca4f80ab50fb0b"} Oct 30 23:59:45.538676 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:45.538656 5073 scope.go:122] "RemoveContainer" containerID="2057f0b41c2e1145df4c43efe708098fed2cd2168d2d4beacbca4f80ab50fb0b" Oct 30 23:59:46.544280 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:46.544243 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kz5pz" event={"ID":"a2587ab9-e842-47c6-87b5-fb0b1ab8b76e","Type":"ContainerStarted","Data":"e9f8d77dfaa89173aacaeeb20f7b2c935f330bb7c45596e44a20240ab16eea54"} Oct 30 23:59:52.522297 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.522255 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-98c775b85-mpvpw" podUID="29d12cf1-c32e-4e32-8df3-8221274ee956" containerName="console" containerID="cri-o://203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21" gracePeriod=15 Oct 30 23:59:52.569152 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.569126 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:52.569259 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.569183 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:52.574158 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.574139 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:52.802197 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.802130 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:59:52.867375 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867349 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867493 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867385 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867493 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867402 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867493 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867441 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867493 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867477 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867514 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867549 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l\" (UniqueName: \"kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867584 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config\") pod \"29d12cf1-c32e-4e32-8df3-8221274ee956\" (UID: \"29d12cf1-c32e-4e32-8df3-8221274ee956\") " Oct 30 23:59:52.867679 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867654 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:52.867929 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867911 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d12cf1-c32e-4e32-8df3-8221274ee956-tmp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.867991 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867918 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:52.867991 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867947 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:52.868113 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.867997 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:52.868226 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.868201 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:52.870197 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.870170 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:52.870506 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.870480 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:52.870594 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.870513 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l" pod "29d12cf1-c32e-4e32-8df3-8221274ee956" (UID: "29d12cf1-c32e-4e32-8df3-8221274ee956"). InnerVolumeSpecName "kube-api-access-bpv2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:52.968802 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968763 5073 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-oauth-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.968900 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968812 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-trusted-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.968900 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968823 5073 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-oauth-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.968900 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968832 5073 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29d12cf1-c32e-4e32-8df3-8221274ee956-console-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.968900 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968841 5073 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-console-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.968900 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968849 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29d12cf1-c32e-4e32-8df3-8221274ee956-service-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:52.968900 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:52.968857 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpv2l\" (UniqueName: \"kubernetes.io/projected/29d12cf1-c32e-4e32-8df3-8221274ee956-kube-api-access-bpv2l\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:53.575648 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.575614 5073 generic.go:356] "Generic (PLEG): container finished" podID="29d12cf1-c32e-4e32-8df3-8221274ee956" containerID="203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21" exitCode=0 Oct 30 23:59:53.576081 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.575664 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98c775b85-mpvpw" Oct 30 23:59:53.576081 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.575699 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98c775b85-mpvpw" event={"ID":"29d12cf1-c32e-4e32-8df3-8221274ee956","Type":"ContainerDied","Data":"203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21"} Oct 30 23:59:53.576081 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.575753 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98c775b85-mpvpw" event={"ID":"29d12cf1-c32e-4e32-8df3-8221274ee956","Type":"ContainerDied","Data":"80b0027270fcb8a3033d520d9b79b24f68f0a38ab4b603e38dc009c2ae42a18d"} Oct 30 23:59:53.576081 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.575772 5073 scope.go:122] "RemoveContainer" containerID="203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21" Oct 30 23:59:53.580572 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.580553 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bdd949fc8-ct86f" Oct 30 23:59:53.585203 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.585187 5073 scope.go:122] "RemoveContainer" containerID="203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21" Oct 30 23:59:53.585473 ip-10-0-140-246 kubenswrapper[5073]: E1030 23:59:53.585455 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21\": container with ID starting with 203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21 not found: ID does not exist" containerID="203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21" Oct 30 23:59:53.585520 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.585480 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21"} err="failed to get container status \"203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21\": rpc error: code = NotFound desc = could not find container \"203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21\": container with ID starting with 203579eab61c79b1b3ce951108979fad25c64a3f889a397feb40af7e76c1ff21 not found: ID does not exist" Oct 30 23:59:53.619926 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.619903 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-98c775b85-mpvpw"] Oct 30 23:59:53.624412 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.624391 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-98c775b85-mpvpw"] Oct 30 23:59:53.637015 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:53.636995 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777cbb6f9c-tzl4n"] Oct 30 23:59:54.059200 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:54.059169 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d12cf1-c32e-4e32-8df3-8221274ee956" path="/var/lib/kubelet/pods/29d12cf1-c32e-4e32-8df3-8221274ee956/volumes" Oct 30 23:59:54.583159 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:54.583124 5073 generic.go:356] "Generic (PLEG): container finished" podID="d91a42d7-9685-4f8f-b620-5e93d58b1a76" containerID="30baadfeac507322a135f2667e34b9f22cdd354c91289a720aa1356301e3e0cf" exitCode=0 Oct 30 23:59:54.583549 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:54.583197 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bbltk-htq8x" event={"ID":"d91a42d7-9685-4f8f-b620-5e93d58b1a76","Type":"ContainerDied","Data":"30baadfeac507322a135f2667e34b9f22cdd354c91289a720aa1356301e3e0cf"} Oct 30 23:59:55.745093 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.745070 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:59:55.790954 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.790924 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") pod \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " Oct 30 23:59:55.791127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791011 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") pod \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " Oct 30 23:59:55.791127 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791056 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") pod \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " Oct 30 23:59:55.791256 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791179 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") pod \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\" (UID: \"d91a42d7-9685-4f8f-b620-5e93d58b1a76\") " Oct 30 23:59:55.791313 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791291 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path" pod "d91a42d7-9685-4f8f-b620-5e93d58b1a76" (UID: "d91a42d7-9685-4f8f-b620-5e93d58b1a76"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:55.791367 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791335 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle" pod "d91a42d7-9685-4f8f-b620-5e93d58b1a76" (UID: "d91a42d7-9685-4f8f-b620-5e93d58b1a76"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:55.791544 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791518 5073 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d91a42d7-9685-4f8f-b620-5e93d58b1a76-archives-path\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:55.791544 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791545 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-service-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:55.791907 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.791599 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle" pod "d91a42d7-9685-4f8f-b620-5e93d58b1a76" (UID: "d91a42d7-9685-4f8f-b620-5e93d58b1a76"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:55.793375 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.793357 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z" pod "d91a42d7-9685-4f8f-b620-5e93d58b1a76" (UID: "d91a42d7-9685-4f8f-b620-5e93d58b1a76"). InnerVolumeSpecName "kube-api-access-dnw7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:55.892481 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.892429 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnw7z\" (UniqueName: \"kubernetes.io/projected/d91a42d7-9685-4f8f-b620-5e93d58b1a76-kube-api-access-dnw7z\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:55.892481 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:55.892453 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91a42d7-9685-4f8f-b620-5e93d58b1a76-trusted-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:56.593671 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:56.593642 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bbltk-htq8x" Oct 30 23:59:56.593671 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:56.593665 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bbltk-htq8x" event={"ID":"d91a42d7-9685-4f8f-b620-5e93d58b1a76","Type":"ContainerDied","Data":"14678713d30ae56b8d949a1122f2428e6956a5d14af1a67d12a4fb48d42860b5"} Oct 30 23:59:56.593892 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:56.593707 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14678713d30ae56b8d949a1122f2428e6956a5d14af1a67d12a4fb48d42860b5" Oct 30 23:59:59.606718 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:59.606691 5073 generic.go:356] "Generic (PLEG): container finished" podID="6afa2442-496b-45d4-ae52-133e0d21a60f" containerID="2e2741d7dd465702f282be021274e55f4269f2f453ec8296a332791204090a82" exitCode=0 Oct 30 23:59:59.607097 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:59.606775 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" event={"ID":"6afa2442-496b-45d4-ae52-133e0d21a60f","Type":"ContainerDied","Data":"2e2741d7dd465702f282be021274e55f4269f2f453ec8296a332791204090a82"} Oct 30 23:59:59.607137 ip-10-0-140-246 kubenswrapper[5073]: I1030 23:59:59.607093 5073 scope.go:122] "RemoveContainer" containerID="2e2741d7dd465702f282be021274e55f4269f2f453ec8296a332791204090a82" Oct 31 00:00:00.612878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:00.612839 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc" event={"ID":"6afa2442-496b-45d4-ae52-133e0d21a60f","Type":"ContainerStarted","Data":"58e7d57f67018f2d2356038d57a48dec711c4ffff02e58dac3bae956774b2bd6"} Oct 31 00:00:05.636105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:05.636075 5073 generic.go:356] "Generic (PLEG): container finished" podID="beff8893-2560-45f8-a96d-bce1e51dcf80" containerID="f0c4a8ce8962645de603ea9cb8ce4c91dfaafedb5c1314b29f3c08e5cc5f3c7d" exitCode=0 Oct 31 00:00:05.636473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:05.636147 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" event={"ID":"beff8893-2560-45f8-a96d-bce1e51dcf80","Type":"ContainerDied","Data":"f0c4a8ce8962645de603ea9cb8ce4c91dfaafedb5c1314b29f3c08e5cc5f3c7d"} Oct 31 00:00:05.636473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:05.636428 5073 scope.go:122] "RemoveContainer" containerID="f0c4a8ce8962645de603ea9cb8ce4c91dfaafedb5c1314b29f3c08e5cc5f3c7d" Oct 31 00:00:06.109803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.109751 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bhj77-m945q"] Oct 31 00:00:06.110118 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.110106 5073 cpu_manager.go:414] "containerMap: removing container" podUID="29d12cf1-c32e-4e32-8df3-8221274ee956" containerName="console" Oct 31 00:00:06.110161 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.110120 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d12cf1-c32e-4e32-8df3-8221274ee956" containerName="console" Oct 31 00:00:06.110161 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.110136 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d91a42d7-9685-4f8f-b620-5e93d58b1a76" containerName="insights-gathering" Oct 31 00:00:06.110161 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.110142 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91a42d7-9685-4f8f-b620-5e93d58b1a76" containerName="insights-gathering" Oct 31 00:00:06.110254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.110193 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d12cf1-c32e-4e32-8df3-8221274ee956" containerName="console" Oct 31 00:00:06.110254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.110201 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91a42d7-9685-4f8f-b620-5e93d58b1a76" containerName="insights-gathering" Oct 31 00:00:06.112084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.112069 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.116559 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.116539 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6l954\"" Oct 31 00:00:06.122842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.122820 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bhj77-m945q"] Oct 31 00:00:06.164985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.164959 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.164985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.164992 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.165163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.165027 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8jq\" (UniqueName: \"kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.165200 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.165167 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.266342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.266298 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.266524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.266372 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.266524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.266393 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.266524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.266423 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8jq\" (UniqueName: \"kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.266875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.266851 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.267014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.266996 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.267145 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.267128 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.281853 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.281828 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8jq\" (UniqueName: \"kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq\") pod \"periodic-gathering-bhj77-m945q\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.423123 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.423038 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:00:06.642761 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.642722 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-nnh9c" event={"ID":"beff8893-2560-45f8-a96d-bce1e51dcf80","Type":"ContainerStarted","Data":"ff5c9df6fe3245a8716fcef9d7e7a4f062271f73707e86f60b70d6902ef68f18"} Oct 31 00:00:06.701248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:06.701176 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bhj77-m945q"] Oct 31 00:00:06.704622 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:00:06.704592 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3d2e09_a636_401f_9852_3a74004cfc1b.slice/crio-4e6a2dcc0b282d3cb289e14cf325ea2a9e87534c014b15ed599effcd974ddbb6 WatchSource:0}: Error finding container 4e6a2dcc0b282d3cb289e14cf325ea2a9e87534c014b15ed599effcd974ddbb6: Status 404 returned error can't find the container with id 4e6a2dcc0b282d3cb289e14cf325ea2a9e87534c014b15ed599effcd974ddbb6 Oct 31 00:00:07.649304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:07.649263 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhj77-m945q" event={"ID":"bf3d2e09-a636-401f-9852-3a74004cfc1b","Type":"ContainerStarted","Data":"33c29936a2ac5c013e74c9749b8ac41607055b3ba55a5d5b1556856f5d9df73e"} Oct 31 00:00:07.649713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:07.649325 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhj77-m945q" event={"ID":"bf3d2e09-a636-401f-9852-3a74004cfc1b","Type":"ContainerStarted","Data":"4e6a2dcc0b282d3cb289e14cf325ea2a9e87534c014b15ed599effcd974ddbb6"} Oct 31 00:00:07.668342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:07.668292 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bhj77-m945q" podStartSLOduration=1.6682791639999999 podStartE2EDuration="1.668279164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:07.667348751 +0000 UTC m=+91.728123471" watchObservedRunningTime="2026-10-31 00:00:07.668279164 +0000 UTC m=+91.729053860" Oct 31 00:00:13.084391 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:00:13.084357 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:00:18.659484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.659439 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-777cbb6f9c-tzl4n" podUID="b9844dc0-181b-4b78-8fcb-c50586b024b9" containerName="console" containerID="cri-o://be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4" gracePeriod=15 Oct 31 00:00:18.910497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.910444 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 31 00:00:18.962926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.962902 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.962935 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2\" (UniqueName: \"kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.962984 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963154 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963215 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963263 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963264 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.963390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963310 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963351 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert\") pod \"b9844dc0-181b-4b78-8fcb-c50586b024b9\" (UID: \"b9844dc0-181b-4b78-8fcb-c50586b024b9\") " Oct 31 00:00:18.963485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963435 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:00:18.963622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963590 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.963763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963675 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.963763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963721 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-service-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.963763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963737 5073 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.963763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963749 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9844dc0-181b-4b78-8fcb-c50586b024b9-tmp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:18.963988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.963936 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:18.965419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.965393 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "kube-api-access-skdl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:18.965507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.965449 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:18.965547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:18.965530 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert" pod "b9844dc0-181b-4b78-8fcb-c50586b024b9" (UID: "b9844dc0-181b-4b78-8fcb-c50586b024b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:19.064590 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.064565 5073 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-oauth-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.064590 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.064586 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-trusted-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.064719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.064597 5073 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9844dc0-181b-4b78-8fcb-c50586b024b9-oauth-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.064719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.064606 5073 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9844dc0-181b-4b78-8fcb-c50586b024b9-console-serving-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.064719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.064615 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdl2\" (UniqueName: \"kubernetes.io/projected/b9844dc0-181b-4b78-8fcb-c50586b024b9-kube-api-access-skdl2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:19.698912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.698874 5073 generic.go:356] "Generic (PLEG): container finished" podID="b9844dc0-181b-4b78-8fcb-c50586b024b9" containerID="be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4" exitCode=0 Oct 31 00:00:19.699311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.698927 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777cbb6f9c-tzl4n" Oct 31 00:00:19.699311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.698957 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777cbb6f9c-tzl4n" event={"ID":"b9844dc0-181b-4b78-8fcb-c50586b024b9","Type":"ContainerDied","Data":"be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4"} Oct 31 00:00:19.699311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.699012 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777cbb6f9c-tzl4n" event={"ID":"b9844dc0-181b-4b78-8fcb-c50586b024b9","Type":"ContainerDied","Data":"2ba186af7a761e98e05ccaac742738db47a4205381eb7f4c3f3ea34295a2ff47"} Oct 31 00:00:19.699311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.699032 5073 scope.go:122] "RemoveContainer" containerID="be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4" Oct 31 00:00:19.712871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.712849 5073 scope.go:122] "RemoveContainer" containerID="be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4" Oct 31 00:00:19.713137 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:00:19.713112 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4\": container with ID starting with be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4 not found: ID does not exist" containerID="be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4" Oct 31 00:00:19.713214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.713144 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4"} err="failed to get container status \"be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4\": rpc error: code = NotFound desc = could not find container \"be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4\": container with ID starting with be642eb7b182127f42484df1c0ea04b89b35f43477400703394ebad2ae891de4 not found: ID does not exist" Oct 31 00:00:19.729834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.729810 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777cbb6f9c-tzl4n"] Oct 31 00:00:19.731635 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:19.731616 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-777cbb6f9c-tzl4n"] Oct 31 00:00:20.058431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:00:20.058404 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9844dc0-181b-4b78-8fcb-c50586b024b9" path="/var/lib/kubelet/pods/b9844dc0-181b-4b78-8fcb-c50586b024b9/volumes" Oct 31 00:00:43.088151 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:00:43.088112 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:01:13.086358 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:01:13.086282 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:01:30.997467 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:30.997435 5073 generic.go:356] "Generic (PLEG): container finished" podID="bf3d2e09-a636-401f-9852-3a74004cfc1b" containerID="33c29936a2ac5c013e74c9749b8ac41607055b3ba55a5d5b1556856f5d9df73e" exitCode=0 Oct 31 00:01:30.997874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:30.997480 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhj77-m945q" event={"ID":"bf3d2e09-a636-401f-9852-3a74004cfc1b","Type":"ContainerDied","Data":"33c29936a2ac5c013e74c9749b8ac41607055b3ba55a5d5b1556856f5d9df73e"} Oct 31 00:01:32.131811 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.131766 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:01:32.199719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.199690 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle\") pod \"bf3d2e09-a636-401f-9852-3a74004cfc1b\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " Oct 31 00:01:32.199865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.199728 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq\" (UniqueName: \"kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq\") pod \"bf3d2e09-a636-401f-9852-3a74004cfc1b\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " Oct 31 00:01:32.199865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.199753 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path\") pod \"bf3d2e09-a636-401f-9852-3a74004cfc1b\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " Oct 31 00:01:32.199937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.199907 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle\") pod \"bf3d2e09-a636-401f-9852-3a74004cfc1b\" (UID: \"bf3d2e09-a636-401f-9852-3a74004cfc1b\") " Oct 31 00:01:32.200179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.200146 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path" pod "bf3d2e09-a636-401f-9852-3a74004cfc1b" (UID: "bf3d2e09-a636-401f-9852-3a74004cfc1b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:01:32.200277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.200184 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle" pod "bf3d2e09-a636-401f-9852-3a74004cfc1b" (UID: "bf3d2e09-a636-401f-9852-3a74004cfc1b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:32.200277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.200237 5073 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/bf3d2e09-a636-401f-9852-3a74004cfc1b-archives-path\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:32.200277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.200252 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle" pod "bf3d2e09-a636-401f-9852-3a74004cfc1b" (UID: "bf3d2e09-a636-401f-9852-3a74004cfc1b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:01:32.201989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.201970 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq" pod "bf3d2e09-a636-401f-9852-3a74004cfc1b" (UID: "bf3d2e09-a636-401f-9852-3a74004cfc1b"). InnerVolumeSpecName "kube-api-access-pw8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:01:32.301324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.301249 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-service-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:32.301324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.301294 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8jq\" (UniqueName: \"kubernetes.io/projected/bf3d2e09-a636-401f-9852-3a74004cfc1b-kube-api-access-pw8jq\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:32.301324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:32.301305 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d2e09-a636-401f-9852-3a74004cfc1b-trusted-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:01:33.007563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:33.007533 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bhj77-m945q" Oct 31 00:01:33.007563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:33.007548 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bhj77-m945q" event={"ID":"bf3d2e09-a636-401f-9852-3a74004cfc1b","Type":"ContainerDied","Data":"4e6a2dcc0b282d3cb289e14cf325ea2a9e87534c014b15ed599effcd974ddbb6"} Oct 31 00:01:33.007770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:01:33.007589 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6a2dcc0b282d3cb289e14cf325ea2a9e87534c014b15ed599effcd974ddbb6" Oct 31 00:01:43.084807 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:01:43.084744 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:02:13.084749 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:02:13.084709 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:02:43.086086 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:02:43.086045 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:03:13.087969 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:03:13.087920 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:03:36.079486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:03:36.079452 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:03:36.081510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:03:36.081483 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:03:36.087862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:03:36.087839 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:03:36.089404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:03:36.089384 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:03:43.088411 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:03:43.088372 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:04:13.085834 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:04:13.085730 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:04:43.085750 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:04:43.085715 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:04:54.156112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156084 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b"] Oct 31 00:04:54.156472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156400 5073 cpu_manager.go:414] "containerMap: removing container" podUID="bf3d2e09-a636-401f-9852-3a74004cfc1b" containerName="insights-gathering" Oct 31 00:04:54.156472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156411 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3d2e09-a636-401f-9852-3a74004cfc1b" containerName="insights-gathering" Oct 31 00:04:54.156472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156423 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b9844dc0-181b-4b78-8fcb-c50586b024b9" containerName="console" Oct 31 00:04:54.156472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156430 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9844dc0-181b-4b78-8fcb-c50586b024b9" containerName="console" Oct 31 00:04:54.156605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156502 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3d2e09-a636-401f-9852-3a74004cfc1b" containerName="insights-gathering" Oct 31 00:04:54.156605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.156512 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9844dc0-181b-4b78-8fcb-c50586b024b9" containerName="console" Oct 31 00:04:54.159430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.159411 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.164432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.164404 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:04:54.164554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.164455 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 31 00:04:54.172890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.172870 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b"] Oct 31 00:04:54.220890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.220860 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-dbkp4\"" Oct 31 00:04:54.220998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.220907 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 31 00:04:54.291852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.291829 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.291947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.291872 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.291947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.291892 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99lc\" (UniqueName: \"kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.392960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.392935 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.393066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.392972 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.393066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.392997 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99lc\" (UniqueName: \"kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.395542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.395522 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.395542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.395532 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.406468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.406416 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99lc\" (UniqueName: \"kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc\") pod \"opentelemetry-operator-webhook-5b9f6959f4-zzz9b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.471020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.470997 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:54.755260 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.755229 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b"] Oct 31 00:04:54.759375 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:04:54.759344 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d41538_45c3_42f9_b6ab_9ad919b4c97b.slice/crio-198bddad1719fefda521865ead038082d31ba98c26987d5a42f3869e288f2ebd WatchSource:0}: Error finding container 198bddad1719fefda521865ead038082d31ba98c26987d5a42f3869e288f2ebd: Status 404 returned error can't find the container with id 198bddad1719fefda521865ead038082d31ba98c26987d5a42f3869e288f2ebd Oct 31 00:04:54.761741 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.761724 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:04:54.834776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:54.834743 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" event={"ID":"97d41538-45c3-42f9-b6ab-9ad919b4c97b","Type":"ContainerStarted","Data":"198bddad1719fefda521865ead038082d31ba98c26987d5a42f3869e288f2ebd"} Oct 31 00:04:56.845316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:56.845289 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" event={"ID":"97d41538-45c3-42f9-b6ab-9ad919b4c97b","Type":"ContainerStarted","Data":"e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9"} Oct 31 00:04:56.845744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:56.845427 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:56.872370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:56.872314 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" podStartSLOduration=0.860857793 podStartE2EDuration="2.872301938s" totalImagesPullingTime="2.011444145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:56.87094258 +0000 UTC m=+380.931717275" watchObservedRunningTime="2026-10-31 00:04:56.872301938 +0000 UTC m=+380.933076633" Oct 31 00:04:57.143253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:57.143182 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b"] Oct 31 00:04:58.853979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:58.853929 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" podUID="97d41538-45c3-42f9-b6ab-9ad919b4c97b" containerName="webhook" containerID="cri-o://e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9" gracePeriod=30 Oct 31 00:04:59.102138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.102115 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:59.232190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.232161 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert\") pod \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " Oct 31 00:04:59.232356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.232206 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert\") pod \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " Oct 31 00:04:59.232356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.232253 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc\" (UniqueName: \"kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc\") pod \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\" (UID: \"97d41538-45c3-42f9-b6ab-9ad919b4c97b\") " Oct 31 00:04:59.234425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.234391 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert" pod "97d41538-45c3-42f9-b6ab-9ad919b4c97b" (UID: "97d41538-45c3-42f9-b6ab-9ad919b4c97b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:59.234537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.234437 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc" pod "97d41538-45c3-42f9-b6ab-9ad919b4c97b" (UID: "97d41538-45c3-42f9-b6ab-9ad919b4c97b"). InnerVolumeSpecName "kube-api-access-x99lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:59.234537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.234484 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert" pod "97d41538-45c3-42f9-b6ab-9ad919b4c97b" (UID: "97d41538-45c3-42f9-b6ab-9ad919b4c97b"). InnerVolumeSpecName "apiservice-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:59.333615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.333589 5073 reconciler_common.go:299] "Volume detached for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-apiservice-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:59.333615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.333611 5073 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d41538-45c3-42f9-b6ab-9ad919b4c97b-webhook-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:59.333754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.333621 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x99lc\" (UniqueName: \"kubernetes.io/projected/97d41538-45c3-42f9-b6ab-9ad919b4c97b-kube-api-access-x99lc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:59.859205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.859174 5073 generic.go:356] "Generic (PLEG): container finished" podID="97d41538-45c3-42f9-b6ab-9ad919b4c97b" containerID="e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9" exitCode=0 Oct 31 00:04:59.859580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.859225 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" Oct 31 00:04:59.859580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.859258 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" event={"ID":"97d41538-45c3-42f9-b6ab-9ad919b4c97b","Type":"ContainerDied","Data":"e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9"} Oct 31 00:04:59.859580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.859315 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b" event={"ID":"97d41538-45c3-42f9-b6ab-9ad919b4c97b","Type":"ContainerDied","Data":"198bddad1719fefda521865ead038082d31ba98c26987d5a42f3869e288f2ebd"} Oct 31 00:04:59.859580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.859341 5073 scope.go:122] "RemoveContainer" containerID="e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9" Oct 31 00:04:59.868206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.868190 5073 scope.go:122] "RemoveContainer" containerID="e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9" Oct 31 00:04:59.868522 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:04:59.868503 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9\": container with ID starting with e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9 not found: ID does not exist" containerID="e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9" Oct 31 00:04:59.868587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.868530 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9"} err="failed to get container status \"e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9\": rpc error: code = NotFound desc = could not find container \"e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9\": container with ID starting with e6dfada80100c718f81c09288c77541f7ddd9088eac7fea0b059737bdc01fdb9 not found: ID does not exist" Oct 31 00:04:59.884920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.884896 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b"] Oct 31 00:04:59.891540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:04:59.891520 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5b9f6959f4-zzz9b"] Oct 31 00:05:00.058074 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:00.058052 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d41538-45c3-42f9-b6ab-9ad919b4c97b" path="/var/lib/kubelet/pods/97d41538-45c3-42f9-b6ab-9ad919b4c97b/volumes" Oct 31 00:05:11.208621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.208585 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl"] Oct 31 00:05:11.209116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.209096 5073 cpu_manager.go:414] "containerMap: removing container" podUID="97d41538-45c3-42f9-b6ab-9ad919b4c97b" containerName="webhook" Oct 31 00:05:11.209176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.209121 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d41538-45c3-42f9-b6ab-9ad919b4c97b" containerName="webhook" Oct 31 00:05:11.209228 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.209216 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d41538-45c3-42f9-b6ab-9ad919b4c97b" containerName="webhook" Oct 31 00:05:11.214029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.214012 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.219710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.219690 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-controller-manager-service-cert\"" Oct 31 00:05:11.220603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.220368 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators-redhat\"/\"kube-root-ca.crt\"" Oct 31 00:05:11.220603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.220467 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-manager-config\"" Oct 31 00:05:11.220603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.220496 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-controller-manager-dockercfg-wvw8q\"" Oct 31 00:05:11.220603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.220520 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators-redhat\"/\"openshift-service-ca.crt\"" Oct 31 00:05:11.220603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.220573 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-metrics\"" Oct 31 00:05:11.224418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.224395 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl"] Oct 31 00:05:11.322242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.322215 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"loki-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-loki-operator-metrics-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.322378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.322261 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-manager-config\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.322378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.322287 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-apiservice-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.322378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.322358 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh24h\" (UniqueName: \"kubernetes.io/projected/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-kube-api-access-xh24h\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.322537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.322415 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-webhook-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.423659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.423624 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-webhook-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.423769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.423670 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"loki-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-loki-operator-metrics-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.423769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.423710 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-manager-config\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.423769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.423735 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-apiservice-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.423769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.423751 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh24h\" (UniqueName: \"kubernetes.io/projected/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-kube-api-access-xh24h\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.424480 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.424410 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-manager-config\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.426214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.426190 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-webhook-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.426214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.426203 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"loki-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-loki-operator-metrics-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.426383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.426297 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-apiservice-cert\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.437842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.437822 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh24h\" (UniqueName: \"kubernetes.io/projected/c8f643e2-9c77-4eb6-a0bd-bc9c99033732-kube-api-access-xh24h\") pod \"loki-operator-controller-manager-5666dc7fd5-lfnkl\" (UID: \"c8f643e2-9c77-4eb6-a0bd-bc9c99033732\") " pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.534268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.534237 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:11.816100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.816029 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl"] Oct 31 00:05:11.820296 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:05:11.820269 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f643e2_9c77_4eb6_a0bd_bc9c99033732.slice/crio-fdae1134b02450107bd3d930682b43b0381872b3f934133aff5e925813273763 WatchSource:0}: Error finding container fdae1134b02450107bd3d930682b43b0381872b3f934133aff5e925813273763: Status 404 returned error can't find the container with id fdae1134b02450107bd3d930682b43b0381872b3f934133aff5e925813273763 Oct 31 00:05:11.915146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:11.915116 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" event={"ID":"c8f643e2-9c77-4eb6-a0bd-bc9c99033732","Type":"ContainerStarted","Data":"fdae1134b02450107bd3d930682b43b0381872b3f934133aff5e925813273763"} Oct 31 00:05:13.085097 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:05:13.085063 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:05:21.962315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:21.962265 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" event={"ID":"c8f643e2-9c77-4eb6-a0bd-bc9c99033732","Type":"ContainerStarted","Data":"fb29696b629384114c4935e1bc123dfdc160fcfc86de574d7833a23f322810c3"} Oct 31 00:05:21.962685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:21.962330 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:21.991657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:21.991586 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" podStartSLOduration=1.916020692 podStartE2EDuration="10.991571797s" totalImagesPullingTime="9.075551105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:21.990114524 +0000 UTC m=+406.050889219" watchObservedRunningTime="2026-10-31 00:05:21.991571797 +0000 UTC m=+406.052346491" Oct 31 00:05:32.968720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:05:32.968681 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators-redhat/loki-operator-controller-manager-5666dc7fd5-lfnkl" Oct 31 00:05:43.084460 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:05:43.084419 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:06:13.084800 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:13.084753 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:06:16.792849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.792779 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:16.796221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.796198 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:16.800360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.800338 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:06:16.801367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.801344 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:06:16.801594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.801572 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:16.801703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.801656 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:16.801760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.801716 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:06:16.812241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.812213 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:16.905730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.905705 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:16.905907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.905759 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvxj\" (UniqueName: \"kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:16.905907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.905883 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:16.905996 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:16.905916 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.007100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.007072 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.007230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.007118 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.007230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.007138 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.007230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.007205 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvxj\" (UniqueName: \"kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.007841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.007819 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.009645 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.009621 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.009765 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.009743 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.020396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.020370 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvxj\" (UniqueName: \"kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj\") pod \"data-science-collector-collector-0\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.112946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.112882 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:17.392775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:17.392749 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:17.395248 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:06:17.395223 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f828d4a_c9bb_47bf_9e4e_207e38e65eae.slice/crio-36fc2c497bfa279ac2d729026efb88181f1428e0e558573c31c4c851793b7b67 WatchSource:0}: Error finding container 36fc2c497bfa279ac2d729026efb88181f1428e0e558573c31c4c851793b7b67: Status 404 returned error can't find the container with id 36fc2c497bfa279ac2d729026efb88181f1428e0e558573c31c4c851793b7b67 Oct 31 00:06:18.204198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:18.204161 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8f828d4a-c9bb-47bf-9e4e-207e38e65eae","Type":"ContainerStarted","Data":"36fc2c497bfa279ac2d729026efb88181f1428e0e558573c31c4c851793b7b67"} Oct 31 00:06:21.776545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:21.776510 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:06:21.776880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:21.776575 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:06:22.224325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:22.224238 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8f828d4a-c9bb-47bf-9e4e-207e38e65eae","Type":"ContainerStarted","Data":"5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31"} Oct 31 00:06:22.244573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:22.244522 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.865832337 podStartE2EDuration="6.244508933s" totalImagesPullingTime="4.378676596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:22.243919843 +0000 UTC m=+466.304694551" watchObservedRunningTime="2026-10-31 00:06:22.244508933 +0000 UTC m=+466.305283627" Oct 31 00:06:34.572356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.572323 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:34.572843 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.572571 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="8f828d4a-c9bb-47bf-9e4e-207e38e65eae" containerName="otc-container" containerID="cri-o://5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31" gracePeriod=30 Oct 31 00:06:34.836017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.835955 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:34.948647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.948623 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal\") pod \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " Oct 31 00:06:34.948760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.948694 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj\" (UniqueName: \"kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj\") pod \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " Oct 31 00:06:34.948905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.948878 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs\") pod \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " Oct 31 00:06:34.949002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.948983 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs\") pod \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\" (UID: \"8f828d4a-c9bb-47bf-9e4e-207e38e65eae\") " Oct 31 00:06:34.949045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.949024 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal" pod "8f828d4a-c9bb-47bf-9e4e-207e38e65eae" (UID: "8f828d4a-c9bb-47bf-9e4e-207e38e65eae"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:34.949298 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.949268 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:34.951140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.951104 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs" pod "8f828d4a-c9bb-47bf-9e4e-207e38e65eae" (UID: "8f828d4a-c9bb-47bf-9e4e-207e38e65eae"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:34.951140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.951120 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs" pod "8f828d4a-c9bb-47bf-9e4e-207e38e65eae" (UID: "8f828d4a-c9bb-47bf-9e4e-207e38e65eae"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:34.951140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:34.951128 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj" pod "8f828d4a-c9bb-47bf-9e4e-207e38e65eae" (UID: "8f828d4a-c9bb-47bf-9e4e-207e38e65eae"). InnerVolumeSpecName "kube-api-access-2jvxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:35.049890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.049858 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvxj\" (UniqueName: \"kubernetes.io/projected/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-kube-api-access-2jvxj\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.049890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.049890 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.049890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.049901 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f828d4a-c9bb-47bf-9e4e-207e38e65eae-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.280098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.280073 5073 generic.go:356] "Generic (PLEG): container finished" podID="8f828d4a-c9bb-47bf-9e4e-207e38e65eae" containerID="5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31" exitCode=0 Oct 31 00:06:35.280221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.280120 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:35.280221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.280158 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8f828d4a-c9bb-47bf-9e4e-207e38e65eae","Type":"ContainerDied","Data":"5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31"} Oct 31 00:06:35.280221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.280211 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8f828d4a-c9bb-47bf-9e4e-207e38e65eae","Type":"ContainerDied","Data":"36fc2c497bfa279ac2d729026efb88181f1428e0e558573c31c4c851793b7b67"} Oct 31 00:06:35.280318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.280232 5073 scope.go:122] "RemoveContainer" containerID="5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31" Oct 31 00:06:35.289296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.289283 5073 scope.go:122] "RemoveContainer" containerID="5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31" Oct 31 00:06:35.289517 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:35.289502 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31\": container with ID starting with 5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31 not found: ID does not exist" containerID="5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31" Oct 31 00:06:35.289561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.289525 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31"} err="failed to get container status \"5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31\": rpc error: code = NotFound desc = could not find container \"5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31\": container with ID starting with 5a192c15043100038315c481f4132c7498d93f01d6e6e066784a0386d699be31 not found: ID does not exist" Oct 31 00:06:35.302602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.302580 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:35.308517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:35.308499 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:36.060833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:36.060803 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f828d4a-c9bb-47bf-9e4e-207e38e65eae" path="/var/lib/kubelet/pods/8f828d4a-c9bb-47bf-9e4e-207e38e65eae/volumes" Oct 31 00:06:38.491666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.491623 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:38.492241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.492220 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8f828d4a-c9bb-47bf-9e4e-207e38e65eae" containerName="otc-container" Oct 31 00:06:38.492304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.492245 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f828d4a-c9bb-47bf-9e4e-207e38e65eae" containerName="otc-container" Oct 31 00:06:38.492356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.492344 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f828d4a-c9bb-47bf-9e4e-207e38e65eae" containerName="otc-container" Oct 31 00:06:38.530447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.530401 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:38.530585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.530539 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.534892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.534724 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:06:38.534892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.534810 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:06:38.536269 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.536245 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:38.536386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.536309 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-pkxt5\"" Oct 31 00:06:38.536724 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.536705 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:06:38.536870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.536850 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:38.536949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.536741 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-b5f8018a\"" Oct 31 00:06:38.676000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.675966 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.676150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.676010 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.676150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.676104 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97z7\" (UniqueName: \"kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.676246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.676151 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.676246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.676232 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.777335 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.777263 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.777335 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.777327 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.777533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.777364 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.777533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.777404 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.777533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.777456 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97z7\" (UniqueName: \"kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.777533 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:38.777405 5073 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:38.777719 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:38.777580 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert podName:f20fa4ee-5465-4bb5-8e27-c28bf10bdc30 nodeName:}" failed. No retries permitted until 2026-10-31 00:06:39.27755329 +0000 UTC m=+483.338327981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30") : secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:38.778127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.778109 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.780138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.780114 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.780205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.780114 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:38.793671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:38.793652 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97z7\" (UniqueName: \"kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:39.280923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:39.280895 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:39.281104 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:39.281016 5073 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:39.281104 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:39.281067 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert podName:f20fa4ee-5465-4bb5-8e27-c28bf10bdc30 nodeName:}" failed. No retries permitted until 2026-10-31 00:06:40.281053974 +0000 UTC m=+484.341828660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30") : secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:40.288304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:40.288275 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:40.290995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:40.290969 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:40.364771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:40.364748 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:40.641394 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:40.641361 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:40.644074 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:06:40.644050 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20fa4ee_5465_4bb5_8e27_c28bf10bdc30.slice/crio-619eebb7232594222400eb11e3fd77ff6da403bbc1e1496077a6ce4db0f79f19 WatchSource:0}: Error finding container 619eebb7232594222400eb11e3fd77ff6da403bbc1e1496077a6ce4db0f79f19: Status 404 returned error can't find the container with id 619eebb7232594222400eb11e3fd77ff6da403bbc1e1496077a6ce4db0f79f19 Oct 31 00:06:40.646258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:40.646205 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:06:40.646355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:40.646298 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:06:41.305964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:41.305928 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30","Type":"ContainerStarted","Data":"48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55"} Oct 31 00:06:41.306388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:41.305977 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30","Type":"ContainerStarted","Data":"619eebb7232594222400eb11e3fd77ff6da403bbc1e1496077a6ce4db0f79f19"} Oct 31 00:06:41.328308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:41.328264 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.328252617 podStartE2EDuration="3.328252617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:41.326987257 +0000 UTC m=+485.387761951" watchObservedRunningTime="2026-10-31 00:06:41.328252617 +0000 UTC m=+485.389027311" Oct 31 00:06:43.085860 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:43.085774 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:06:52.050134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.050039 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:52.050622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.050291 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" containerName="otc-container" containerID="cri-o://48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55" gracePeriod=30 Oct 31 00:06:52.298830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.298807 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:52.353441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.353372 5073 generic.go:356] "Generic (PLEG): container finished" podID="f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" containerID="48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55" exitCode=0 Oct 31 00:06:52.353441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.353411 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:52.353643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.353454 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30","Type":"ContainerDied","Data":"48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55"} Oct 31 00:06:52.353643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.353517 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30","Type":"ContainerDied","Data":"619eebb7232594222400eb11e3fd77ff6da403bbc1e1496077a6ce4db0f79f19"} Oct 31 00:06:52.353643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.353549 5073 scope.go:122] "RemoveContainer" containerID="48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55" Oct 31 00:06:52.361983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.361965 5073 scope.go:122] "RemoveContainer" containerID="48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55" Oct 31 00:06:52.362234 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:06:52.362215 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55\": container with ID starting with 48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55 not found: ID does not exist" containerID="48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55" Oct 31 00:06:52.362290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.362249 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55"} err="failed to get container status \"48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55\": rpc error: code = NotFound desc = could not find container \"48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55\": container with ID starting with 48c01c5b45fee095a7b1a8621b91b96c955f6933e0df4bb016b96401831a4b55 not found: ID does not exist" Oct 31 00:06:52.384673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.384650 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal\") pod \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " Oct 31 00:06:52.384815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.384689 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs\") pod \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " Oct 31 00:06:52.384815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.384711 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") pod \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " Oct 31 00:06:52.384815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.384737 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs\") pod \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " Oct 31 00:06:52.385076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.385050 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal" pod "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:52.386854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.386828 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert" pod "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.386999 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.386977 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs" pod "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.387057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.387007 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs" pod "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.485570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.485540 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7\" (UniqueName: \"kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7\") pod \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\" (UID: \"f20fa4ee-5465-4bb5-8e27-c28bf10bdc30\") " Oct 31 00:06:52.485846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.485825 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.485846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.485846 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.485971 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.485857 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.485971 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.485866 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.487744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.487723 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7" pod "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" (UID: "f20fa4ee-5465-4bb5-8e27-c28bf10bdc30"). InnerVolumeSpecName "kube-api-access-k97z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:52.586146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.586124 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k97z7\" (UniqueName: \"kubernetes.io/projected/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30-kube-api-access-k97z7\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.676330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.676307 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:52.680704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:52.680683 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:54.057954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.057917 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" path="/var/lib/kubelet/pods/f20fa4ee-5465-4bb5-8e27-c28bf10bdc30/volumes" Oct 31 00:06:54.507224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.507189 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:54.507687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.507672 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" containerName="otc-container" Oct 31 00:06:54.507763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.507689 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" containerName="otc-container" Oct 31 00:06:54.507842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.507770 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20fa4ee-5465-4bb5-8e27-c28bf10bdc30" containerName="otc-container" Oct 31 00:06:54.512839 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.512814 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.517767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.517749 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:54.518319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.518128 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:06:54.518319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.518154 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-c631c892\"" Oct 31 00:06:54.518319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.518188 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:54.518319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.518226 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:06:54.518319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.518300 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wgpbb\"" Oct 31 00:06:54.526183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.526161 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:54.600166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.600137 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29khv\" (UniqueName: \"kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.600290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.600200 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.600290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.600261 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.600363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.600289 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.700904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.700879 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29khv\" (UniqueName: \"kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.701020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.700924 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.701020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.700943 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.701213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.701194 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.701476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.701456 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.703485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.703466 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.703568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.703516 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.731193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.731161 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29khv\" (UniqueName: \"kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv\") pod \"data-science-collector-collector-1\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:54.827252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:54.827196 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:55.105622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:55.105550 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:55.109280 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:06:55.109253 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53300624_40ae_4e63_be8b_5bf354aced5b.slice/crio-9333281459aed6af8830a7fbc3afb25c0c5c9074f76088006ad978dfd0879ef3 WatchSource:0}: Error finding container 9333281459aed6af8830a7fbc3afb25c0c5c9074f76088006ad978dfd0879ef3: Status 404 returned error can't find the container with id 9333281459aed6af8830a7fbc3afb25c0c5c9074f76088006ad978dfd0879ef3 Oct 31 00:06:55.111431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:55.111401 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:06:55.111504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:55.111453 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:06:55.370650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:55.370556 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"53300624-40ae-4e63-be8b-5bf354aced5b","Type":"ContainerStarted","Data":"9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225"} Oct 31 00:06:55.370819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:55.370650 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"53300624-40ae-4e63-be8b-5bf354aced5b","Type":"ContainerStarted","Data":"9333281459aed6af8830a7fbc3afb25c0c5c9074f76088006ad978dfd0879ef3"} Oct 31 00:06:55.391343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:06:55.391298 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.391285911 podStartE2EDuration="1.391285911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:55.389495294 +0000 UTC m=+499.450270029" watchObservedRunningTime="2026-10-31 00:06:55.391285911 +0000 UTC m=+499.452060606" Oct 31 00:07:08.487640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.487601 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:08.488222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.488188 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="53300624-40ae-4e63-be8b-5bf354aced5b" containerName="otc-container" containerID="cri-o://9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225" gracePeriod=30 Oct 31 00:07:08.732764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.732734 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:08.807737 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.807673 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv\" (UniqueName: \"kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv\") pod \"53300624-40ae-4e63-be8b-5bf354aced5b\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " Oct 31 00:07:08.807878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.807738 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal\") pod \"53300624-40ae-4e63-be8b-5bf354aced5b\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " Oct 31 00:07:08.807878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.807775 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs\") pod \"53300624-40ae-4e63-be8b-5bf354aced5b\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " Oct 31 00:07:08.807878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.807821 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs\") pod \"53300624-40ae-4e63-be8b-5bf354aced5b\" (UID: \"53300624-40ae-4e63-be8b-5bf354aced5b\") " Oct 31 00:07:08.808120 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.808096 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal" pod "53300624-40ae-4e63-be8b-5bf354aced5b" (UID: "53300624-40ae-4e63-be8b-5bf354aced5b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:08.809958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.809926 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs" pod "53300624-40ae-4e63-be8b-5bf354aced5b" (UID: "53300624-40ae-4e63-be8b-5bf354aced5b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:08.809958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.809940 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs" pod "53300624-40ae-4e63-be8b-5bf354aced5b" (UID: "53300624-40ae-4e63-be8b-5bf354aced5b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:08.810092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.810023 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv" pod "53300624-40ae-4e63-be8b-5bf354aced5b" (UID: "53300624-40ae-4e63-be8b-5bf354aced5b"). InnerVolumeSpecName "kube-api-access-29khv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:08.908457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.908435 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29khv\" (UniqueName: \"kubernetes.io/projected/53300624-40ae-4e63-be8b-5bf354aced5b-kube-api-access-29khv\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.908457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.908457 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53300624-40ae-4e63-be8b-5bf354aced5b-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.908573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.908468 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.908573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:08.908477 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53300624-40ae-4e63-be8b-5bf354aced5b-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:09.433115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.433082 5073 generic.go:356] "Generic (PLEG): container finished" podID="53300624-40ae-4e63-be8b-5bf354aced5b" containerID="9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225" exitCode=0 Oct 31 00:07:09.433283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.433132 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:09.433283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.433152 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"53300624-40ae-4e63-be8b-5bf354aced5b","Type":"ContainerDied","Data":"9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225"} Oct 31 00:07:09.433283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.433195 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"53300624-40ae-4e63-be8b-5bf354aced5b","Type":"ContainerDied","Data":"9333281459aed6af8830a7fbc3afb25c0c5c9074f76088006ad978dfd0879ef3"} Oct 31 00:07:09.433283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.433216 5073 scope.go:122] "RemoveContainer" containerID="9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225" Oct 31 00:07:09.442273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.442255 5073 scope.go:122] "RemoveContainer" containerID="9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225" Oct 31 00:07:09.442542 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:09.442520 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225\": container with ID starting with 9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225 not found: ID does not exist" containerID="9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225" Oct 31 00:07:09.442608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.442554 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225"} err="failed to get container status \"9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225\": rpc error: code = NotFound desc = could not find container \"9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225\": container with ID starting with 9bfb7a598303152797afe541e3cedccc6d138458b133fe3c41625d36208d3225 not found: ID does not exist" Oct 31 00:07:09.454391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.454369 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:09.459182 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:09.459162 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:10.057696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:10.057666 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53300624-40ae-4e63-be8b-5bf354aced5b" path="/var/lib/kubelet/pods/53300624-40ae-4e63-be8b-5bf354aced5b/volumes" Oct 31 00:07:12.123569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.123534 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:12.124052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.124036 5073 cpu_manager.go:414] "containerMap: removing container" podUID="53300624-40ae-4e63-be8b-5bf354aced5b" containerName="otc-container" Oct 31 00:07:12.124091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.124056 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53300624-40ae-4e63-be8b-5bf354aced5b" containerName="otc-container" Oct 31 00:07:12.124196 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.124185 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="53300624-40ae-4e63-be8b-5bf354aced5b" containerName="otc-container" Oct 31 00:07:12.129098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.129066 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.137471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.137448 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:12.141396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.141372 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-4d267ad2\"" Oct 31 00:07:12.141724 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.141663 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:07:12.141960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.141937 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:07:12.180183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.180158 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:07:12.180403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.180286 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-ssct5\"" Oct 31 00:07:12.180403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.180300 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:07:12.180538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.180397 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:07:12.232622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.232589 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.232796 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.232632 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrqt\" (UniqueName: \"kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.232796 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.232755 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.232907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.232811 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.232974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.232920 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.334029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.334002 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.334164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.334034 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.334164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.334052 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.334164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.334077 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.334164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.334114 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrqt\" (UniqueName: \"kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.334856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.334754 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.336611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.336581 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.336729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.336671 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.336729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.336713 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.346495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.346468 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrqt\" (UniqueName: \"kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt\") pod \"data-science-collector-collector-1\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.446407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.446353 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:12.733625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.733598 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:12.736167 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:07:12.736140 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afc5dc3_923e_43b0_b29c_cf619442aad4.slice/crio-18fa67f0063126b0dc14af3fd9c5f9b49e897d25493777f1f1c4a3ddf1fc418d WatchSource:0}: Error finding container 18fa67f0063126b0dc14af3fd9c5f9b49e897d25493777f1f1c4a3ddf1fc418d: Status 404 returned error can't find the container with id 18fa67f0063126b0dc14af3fd9c5f9b49e897d25493777f1f1c4a3ddf1fc418d Oct 31 00:07:12.738192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.738151 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:07:12.738302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:12.738251 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:07:13.084246 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:13.084167 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:07:13.457063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:13.456980 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2afc5dc3-923e-43b0-b29c-cf619442aad4","Type":"ContainerStarted","Data":"0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727"} Oct 31 00:07:13.457063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:13.457033 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2afc5dc3-923e-43b0-b29c-cf619442aad4","Type":"ContainerStarted","Data":"18fa67f0063126b0dc14af3fd9c5f9b49e897d25493777f1f1c4a3ddf1fc418d"} Oct 31 00:07:25.446156 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.446087 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=13.446068371 podStartE2EDuration="13.446068371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:13.477846287 +0000 UTC m=+517.538620983" watchObservedRunningTime="2026-10-31 00:07:25.446068371 +0000 UTC m=+529.506843070" Oct 31 00:07:25.447694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.447666 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:25.447984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.447952 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2afc5dc3-923e-43b0-b29c-cf619442aad4" containerName="otc-container" containerID="cri-o://0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727" gracePeriod=30 Oct 31 00:07:25.690668 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.690641 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:25.741984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.741958 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert\") pod \"2afc5dc3-923e-43b0-b29c-cf619442aad4\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " Oct 31 00:07:25.742126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.742006 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal\") pod \"2afc5dc3-923e-43b0-b29c-cf619442aad4\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " Oct 31 00:07:25.742126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.742029 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs\") pod \"2afc5dc3-923e-43b0-b29c-cf619442aad4\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " Oct 31 00:07:25.742126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.742077 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs\") pod \"2afc5dc3-923e-43b0-b29c-cf619442aad4\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " Oct 31 00:07:25.742126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.742117 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt\" (UniqueName: \"kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt\") pod \"2afc5dc3-923e-43b0-b29c-cf619442aad4\" (UID: \"2afc5dc3-923e-43b0-b29c-cf619442aad4\") " Oct 31 00:07:25.742394 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.742366 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal" pod "2afc5dc3-923e-43b0-b29c-cf619442aad4" (UID: "2afc5dc3-923e-43b0-b29c-cf619442aad4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:25.744403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.744380 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt" pod "2afc5dc3-923e-43b0-b29c-cf619442aad4" (UID: "2afc5dc3-923e-43b0-b29c-cf619442aad4"). InnerVolumeSpecName "kube-api-access-jkrqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:25.744496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.744408 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs" pod "2afc5dc3-923e-43b0-b29c-cf619442aad4" (UID: "2afc5dc3-923e-43b0-b29c-cf619442aad4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.744496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.744426 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs" pod "2afc5dc3-923e-43b0-b29c-cf619442aad4" (UID: "2afc5dc3-923e-43b0-b29c-cf619442aad4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.744496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.744491 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert" pod "2afc5dc3-923e-43b0-b29c-cf619442aad4" (UID: "2afc5dc3-923e-43b0-b29c-cf619442aad4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.843053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.843027 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.843053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.843050 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2afc5dc3-923e-43b0-b29c-cf619442aad4-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.843194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.843059 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.843194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.843069 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2afc5dc3-923e-43b0-b29c-cf619442aad4-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.843194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:25.843078 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrqt\" (UniqueName: \"kubernetes.io/projected/2afc5dc3-923e-43b0-b29c-cf619442aad4-kube-api-access-jkrqt\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:26.514139 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.514102 5073 generic.go:356] "Generic (PLEG): container finished" podID="2afc5dc3-923e-43b0-b29c-cf619442aad4" containerID="0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727" exitCode=0 Oct 31 00:07:26.514628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.514152 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2afc5dc3-923e-43b0-b29c-cf619442aad4","Type":"ContainerDied","Data":"0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727"} Oct 31 00:07:26.514628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.514177 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:26.514628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.514192 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2afc5dc3-923e-43b0-b29c-cf619442aad4","Type":"ContainerDied","Data":"18fa67f0063126b0dc14af3fd9c5f9b49e897d25493777f1f1c4a3ddf1fc418d"} Oct 31 00:07:26.514628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.514213 5073 scope.go:122] "RemoveContainer" containerID="0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727" Oct 31 00:07:26.523113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.523093 5073 scope.go:122] "RemoveContainer" containerID="0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727" Oct 31 00:07:26.523363 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:26.523344 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727\": container with ID starting with 0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727 not found: ID does not exist" containerID="0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727" Oct 31 00:07:26.523430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.523371 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727"} err="failed to get container status \"0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727\": rpc error: code = NotFound desc = could not find container \"0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727\": container with ID starting with 0d80e2099294a289fb983452aa81e65cf5b098cadb6b1b4dab4d8a0fba215727 not found: ID does not exist" Oct 31 00:07:26.532326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.532306 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:26.537033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:26.537012 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:28.057851 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:28.057814 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afc5dc3-923e-43b0-b29c-cf619442aad4" path="/var/lib/kubelet/pods/2afc5dc3-923e-43b0-b29c-cf619442aad4/volumes" Oct 31 00:07:33.342168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.342137 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw"] Oct 31 00:07:33.342531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.342480 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2afc5dc3-923e-43b0-b29c-cf619442aad4" containerName="otc-container" Oct 31 00:07:33.342531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.342492 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afc5dc3-923e-43b0-b29c-cf619442aad4" containerName="otc-container" Oct 31 00:07:33.342644 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.342567 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afc5dc3-923e-43b0-b29c-cf619442aad4" containerName="otc-container" Oct 31 00:07:33.345729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.345714 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:33.354035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.354010 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw"] Oct 31 00:07:33.355003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.354979 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:07:33.355125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.355054 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:07:33.355183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.355134 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:07:33.405444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.405420 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmkp\" (UniqueName: \"kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw\" (UID: \"12c2b6d3-d63c-4df2-b84e-794b5c524015\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:33.506470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.506441 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmkp\" (UniqueName: \"kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw\" (UID: \"12c2b6d3-d63c-4df2-b84e-794b5c524015\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:33.519251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.519221 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmkp\" (UniqueName: \"kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw\" (UID: \"12c2b6d3-d63c-4df2-b84e-794b5c524015\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:33.658465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.658393 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:33.664192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.664167 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:33.670980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.670958 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.681469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.681367 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:33.683572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.683537 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:07:33.754084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.754053 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz"] Oct 31 00:07:33.757676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.757655 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:33.758815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.758720 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:07:33.758927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.758773 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:07:33.758927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.758825 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:07:33.758927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.758881 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:07:33.758927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.758847 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-6x5wq\"" Oct 31 00:07:33.768556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.768536 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz"] Oct 31 00:07:33.808388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.808355 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.808540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.808404 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.808540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.808511 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.808620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.808548 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.808620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.808595 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dr54\" (UniqueName: \"kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.808685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.808633 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.856322 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.856298 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:07:33.857152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.857133 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-k9fns\"" Oct 31 00:07:33.857211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.857134 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:07:33.909106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909039 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.909106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909078 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.909276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909257 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:33.909337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909292 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.909337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909312 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.909444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909394 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dr54\" (UniqueName: \"kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.909444 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:33.909409 5073 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:07:33.909567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909446 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.909567 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:33.909462 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs podName:2e93e13b-02f7-4f7c-b498-32ec8d0405e1 nodeName:}" failed. No retries permitted until 2026-10-31 00:07:34.409441686 +0000 UTC m=+538.470216375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:07:33.909567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909526 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xcz\" (UniqueName: \"kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:33.909729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.909582 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:33.910401 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.910372 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.911734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.911712 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.911889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.911870 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.911948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.911878 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:33.921962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:33.921943 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dr54\" (UniqueName: \"kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:34.009946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.009911 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xcz\" (UniqueName: \"kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.010078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.009965 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.010078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.010024 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.010637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.010616 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.012500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.012476 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.023437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.023409 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xcz\" (UniqueName: \"kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz\") pod \"data-science-collector-targetallocator-7b577df47b-8v8vz\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.050838 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.050818 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw"] Oct 31 00:07:34.053263 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:07:34.053241 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c2b6d3_d63c_4df2_b84e_794b5c524015.slice/crio-559789dda35139f5a16bde465ee46dd5c4faf45e5fbf5e9f7865db04233cce76 WatchSource:0}: Error finding container 559789dda35139f5a16bde465ee46dd5c4faf45e5fbf5e9f7865db04233cce76: Status 404 returned error can't find the container with id 559789dda35139f5a16bde465ee46dd5c4faf45e5fbf5e9f7865db04233cce76 Oct 31 00:07:34.071616 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.071591 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:34.346620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.346598 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz"] Oct 31 00:07:34.348957 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:07:34.348922 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb82b79c9_bf59_4f71_a076_a1ba614c21eb.slice/crio-6ac840050c0aeea6ab35947e62619e0955270d04e7a4c50a080461584cace7f4 WatchSource:0}: Error finding container 6ac840050c0aeea6ab35947e62619e0955270d04e7a4c50a080461584cace7f4: Status 404 returned error can't find the container with id 6ac840050c0aeea6ab35947e62619e0955270d04e7a4c50a080461584cace7f4 Oct 31 00:07:34.414338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.414314 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:34.416684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.416661 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:34.552707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.552678 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" event={"ID":"b82b79c9-bf59-4f71-a076-a1ba614c21eb","Type":"ContainerStarted","Data":"6ac840050c0aeea6ab35947e62619e0955270d04e7a4c50a080461584cace7f4"} Oct 31 00:07:34.553865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.553839 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" event={"ID":"12c2b6d3-d63c-4df2-b84e-794b5c524015","Type":"ContainerStarted","Data":"559789dda35139f5a16bde465ee46dd5c4faf45e5fbf5e9f7865db04233cce76"} Oct 31 00:07:34.605143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.605095 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:07:34.886312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.886014 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:07:34.892006 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:07:34.891977 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e93e13b_02f7_4f7c_b498_32ec8d0405e1.slice/crio-aa62770a8906bc8a4e158c2f83f3e2cd95568cf98611ff239bc95a66fea0fa02 WatchSource:0}: Error finding container aa62770a8906bc8a4e158c2f83f3e2cd95568cf98611ff239bc95a66fea0fa02: Status 404 returned error can't find the container with id aa62770a8906bc8a4e158c2f83f3e2cd95568cf98611ff239bc95a66fea0fa02 Oct 31 00:07:34.894444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.894408 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:07:34.894529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:34.894468 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:07:35.559321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:35.559288 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e93e13b-02f7-4f7c-b498-32ec8d0405e1","Type":"ContainerStarted","Data":"7d29998c0a3b4628290d4a9fd423e1c8e79a05d8d59c31e25e99d82e37ff2e38"} Oct 31 00:07:35.559708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:35.559351 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e93e13b-02f7-4f7c-b498-32ec8d0405e1","Type":"ContainerStarted","Data":"aa62770a8906bc8a4e158c2f83f3e2cd95568cf98611ff239bc95a66fea0fa02"} Oct 31 00:07:37.276444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.276406 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:07:37.276848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.276479 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:07:37.572358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.572278 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" event={"ID":"b82b79c9-bf59-4f71-a076-a1ba614c21eb","Type":"ContainerStarted","Data":"bc742722a76a27d363efdec067bab139c2b0acf92843d83f4d3c358468bdca14"} Oct 31 00:07:37.572500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.572459 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:37.574393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.574369 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:07:37.592928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.592882 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=4.592866516 podStartE2EDuration="4.592866516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:35.581985438 +0000 UTC m=+539.642760134" watchObservedRunningTime="2026-10-31 00:07:37.592866516 +0000 UTC m=+541.653641213" Oct 31 00:07:37.593847 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:37.593808 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" podStartSLOduration=1.669131065 podStartE2EDuration="4.593793745s" totalImagesPullingTime="2.92466268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:37.592271529 +0000 UTC m=+541.653046230" watchObservedRunningTime="2026-10-31 00:07:37.593793745 +0000 UTC m=+541.654568434" Oct 31 00:07:38.578233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:38.578190 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" event={"ID":"12c2b6d3-d63c-4df2-b84e-794b5c524015","Type":"ContainerStarted","Data":"8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84"} Oct 31 00:07:38.597500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:38.597448 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" podStartSLOduration=1.50689559 podStartE2EDuration="5.597436591s" totalImagesPullingTime="4.090541001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:38.596677608 +0000 UTC m=+542.657452319" watchObservedRunningTime="2026-10-31 00:07:38.597436591 +0000 UTC m=+542.658211277" Oct 31 00:07:39.826486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:39.826450 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d"] Oct 31 00:07:39.830386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:39.830362 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:07:39.838291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:39.838262 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d"] Oct 31 00:07:39.963016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:39.962990 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgpz5\" (UniqueName: \"kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d\" (UID: \"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:07:40.063531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.063503 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgpz5\" (UniqueName: \"kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d\" (UID: \"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:07:40.076863 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.076771 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgpz5\" (UniqueName: \"kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d\" (UID: \"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:07:40.142506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.142473 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:07:40.417615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.417544 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d"] Oct 31 00:07:40.588867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.588834 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" event={"ID":"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7","Type":"ContainerStarted","Data":"273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f"} Oct 31 00:07:40.589006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.588878 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" event={"ID":"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7","Type":"ContainerStarted","Data":"8c4d7a3a81234108b5a828823699227f87b15aa8e8bdda62f37f1c7a97616bc5"} Oct 31 00:07:40.606548 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.606482 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" podStartSLOduration=1.606470516 podStartE2EDuration="1.606470516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:40.605603111 +0000 UTC m=+544.666377806" watchObservedRunningTime="2026-10-31 00:07:40.606470516 +0000 UTC m=+544.667245210" Oct 31 00:07:40.633963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.633938 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw"] Oct 31 00:07:40.634119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.634087 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" podUID="12c2b6d3-d63c-4df2-b84e-794b5c524015" containerName="thanos-querier" containerID="cri-o://8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84" gracePeriod=30 Oct 31 00:07:40.868883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.868857 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:40.971991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.971958 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp\" (UniqueName: \"kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp\") pod \"12c2b6d3-d63c-4df2-b84e-794b5c524015\" (UID: \"12c2b6d3-d63c-4df2-b84e-794b5c524015\") " Oct 31 00:07:40.974076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:40.974054 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp" pod "12c2b6d3-d63c-4df2-b84e-794b5c524015" (UID: "12c2b6d3-d63c-4df2-b84e-794b5c524015"). InnerVolumeSpecName "kube-api-access-vnmkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:41.073479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.073415 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnmkp\" (UniqueName: \"kubernetes.io/projected/12c2b6d3-d63c-4df2-b84e-794b5c524015-kube-api-access-vnmkp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:41.595366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.595330 5073 generic.go:356] "Generic (PLEG): container finished" podID="12c2b6d3-d63c-4df2-b84e-794b5c524015" containerID="8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84" exitCode=0 Oct 31 00:07:41.595550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.595376 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" Oct 31 00:07:41.595550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.595422 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" event={"ID":"12c2b6d3-d63c-4df2-b84e-794b5c524015","Type":"ContainerDied","Data":"8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84"} Oct 31 00:07:41.595550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.595462 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw" event={"ID":"12c2b6d3-d63c-4df2-b84e-794b5c524015","Type":"ContainerDied","Data":"559789dda35139f5a16bde465ee46dd5c4faf45e5fbf5e9f7865db04233cce76"} Oct 31 00:07:41.595550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.595477 5073 scope.go:122] "RemoveContainer" containerID="8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84" Oct 31 00:07:41.604438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.604422 5073 scope.go:122] "RemoveContainer" containerID="8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84" Oct 31 00:07:41.604688 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:41.604672 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84\": container with ID starting with 8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84 not found: ID does not exist" containerID="8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84" Oct 31 00:07:41.604730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.604698 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84"} err="failed to get container status \"8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84\": rpc error: code = NotFound desc = could not find container \"8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84\": container with ID starting with 8bb65ece109abe013f26015f230abd77f3c7d3dd4fc49545e9169dfabb3d8e84 not found: ID does not exist" Oct 31 00:07:41.616614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.616592 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw"] Oct 31 00:07:41.621174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:41.621152 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-q4wzw"] Oct 31 00:07:42.058518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:42.058493 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c2b6d3-d63c-4df2-b84e-794b5c524015" path="/var/lib/kubelet/pods/12c2b6d3-d63c-4df2-b84e-794b5c524015/volumes" Oct 31 00:07:43.085027 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:07:43.084996 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:07:43.550121 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.550086 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:07:43.550561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.550541 5073 cpu_manager.go:414] "containerMap: removing container" podUID="12c2b6d3-d63c-4df2-b84e-794b5c524015" containerName="thanos-querier" Oct 31 00:07:43.550651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.550562 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c2b6d3-d63c-4df2-b84e-794b5c524015" containerName="thanos-querier" Oct 31 00:07:43.550704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.550670 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c2b6d3-d63c-4df2-b84e-794b5c524015" containerName="thanos-querier" Oct 31 00:07:43.555801 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.555760 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.571162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.571134 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:07:43.624232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624193 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:07:43.624232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624206 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:07:43.624429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624237 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:07:43.624429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624244 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:07:43.624429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624240 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:07:43.624429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624287 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:07:43.624429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.624345 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-qtj4l\"" Oct 31 00:07:43.628080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.628064 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:07:43.628163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.628112 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:07:43.628336 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.628322 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:07:43.691076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691053 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691089 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691107 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691159 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691251 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691280 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691303 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691319 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblkc\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691380 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691433 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691461 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.691579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.691483 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.792844 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.792811 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.792844 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.792845 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblkc\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.792864 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793042 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793076 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793105 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793146 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793184 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793302 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793346 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793404 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.793678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.793453 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.794595 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.794463 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.794595 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.794466 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.795427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.795404 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.796264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.796244 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.796264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.796255 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.796390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.796315 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.796430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.796409 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.796543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.796520 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.796593 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.796577 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.797500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.797483 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.798478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.798461 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:43.805815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:43.805750 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblkc\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.321454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.321409 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.329058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.329031 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.422774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.422738 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.422917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.422881 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.426415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.426399 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.599607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.599571 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.615732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.615702 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:47.777861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:47.777820 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:07:48.067731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:48.067707 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:07:48.068845 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:07:48.068819 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa4d324_ae53_4d1b_b3e4_12e520b92266.slice/crio-1955ea28fa12f9aadcf772df2638024099a3bd08ebcdc3f6811a737f9e678932 WatchSource:0}: Error finding container 1955ea28fa12f9aadcf772df2638024099a3bd08ebcdc3f6811a737f9e678932: Status 404 returned error can't find the container with id 1955ea28fa12f9aadcf772df2638024099a3bd08ebcdc3f6811a737f9e678932 Oct 31 00:07:48.630996 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:48.630961 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerStarted","Data":"1955ea28fa12f9aadcf772df2638024099a3bd08ebcdc3f6811a737f9e678932"} Oct 31 00:07:50.640531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:50.640496 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerStarted","Data":"b0b70a80fefb63ba966b4846b80c326d00750157c4983143255e8237e3a5b911"} Oct 31 00:07:53.656799 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:53.656719 5073 generic.go:356] "Generic (PLEG): container finished" podID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerID="b0b70a80fefb63ba966b4846b80c326d00750157c4983143255e8237e3a5b911" exitCode=0 Oct 31 00:07:53.657129 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:53.656810 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerDied","Data":"b0b70a80fefb63ba966b4846b80c326d00750157c4983143255e8237e3a5b911"} Oct 31 00:07:59.689169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:07:59.689099 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerStarted","Data":"9b0f106f670a528080e3650931fdb0601b478ba856ba6a54b5db86227a3f9d23"} Oct 31 00:08:00.695655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:00.695617 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerStarted","Data":"1fdc8d74f8a008d686bec71d38462e3d105f074d7f476d54153d2336a3475b8e"} Oct 31 00:08:00.696165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:00.695730 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerStarted","Data":"050bdb3946d0d7fb6e537be499ca7d448ae299c30b6b8497fd373b5bc193e3c3"} Oct 31 00:08:00.730410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:00.730361 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.075995227 podStartE2EDuration="20.730345482s" totalImagesPullingTime="7.170362585s" totalInitContainerRuntime="3.48398767s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:07:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:00.729106397 +0000 UTC m=+564.789881093" watchObservedRunningTime="2026-10-31 00:08:00.730345482 +0000 UTC m=+564.791120177" Oct 31 00:08:02.778871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:02.778837 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:08:02.779286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:02.778895 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:08:02.784209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:02.784189 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:08:03.714128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:03.714097 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:08:07.305211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.305172 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d"] Oct 31 00:08:07.305686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.305394 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" podUID="712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" containerName="thanos-querier" containerID="cri-o://273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f" gracePeriod=30 Oct 31 00:08:07.401870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.401839 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:08:07.402116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.402090 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="prometheus" containerID="cri-o://9b0f106f670a528080e3650931fdb0601b478ba856ba6a54b5db86227a3f9d23" gracePeriod=600 Oct 31 00:08:07.402197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.402151 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="config-reloader" containerID="cri-o://050bdb3946d0d7fb6e537be499ca7d448ae299c30b6b8497fd373b5bc193e3c3" gracePeriod=600 Oct 31 00:08:07.402261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.402165 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="thanos-sidecar" containerID="cri-o://1fdc8d74f8a008d686bec71d38462e3d105f074d7f476d54153d2336a3475b8e" gracePeriod=600 Oct 31 00:08:07.569161 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.569069 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:08:07.569433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.569404 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2e93e13b-02f7-4f7c-b498-32ec8d0405e1" containerName="otc-container" containerID="cri-o://7d29998c0a3b4628290d4a9fd423e1c8e79a05d8d59c31e25e99d82e37ff2e38" gracePeriod=30 Oct 31 00:08:07.589549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.589519 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz"] Oct 31 00:08:07.589750 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.589723 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" podUID="b82b79c9-bf59-4f71-a076-a1ba614c21eb" containerName="ta-container" containerID="cri-o://bc742722a76a27d363efdec067bab139c2b0acf92843d83f4d3c358468bdca14" gracePeriod=30 Oct 31 00:08:07.731502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.731175 5073 generic.go:356] "Generic (PLEG): container finished" podID="2e93e13b-02f7-4f7c-b498-32ec8d0405e1" containerID="7d29998c0a3b4628290d4a9fd423e1c8e79a05d8d59c31e25e99d82e37ff2e38" exitCode=0 Oct 31 00:08:07.731502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.731271 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e93e13b-02f7-4f7c-b498-32ec8d0405e1","Type":"ContainerDied","Data":"7d29998c0a3b4628290d4a9fd423e1c8e79a05d8d59c31e25e99d82e37ff2e38"} Oct 31 00:08:07.734363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.734224 5073 generic.go:356] "Generic (PLEG): container finished" podID="b82b79c9-bf59-4f71-a076-a1ba614c21eb" containerID="bc742722a76a27d363efdec067bab139c2b0acf92843d83f4d3c358468bdca14" exitCode=0 Oct 31 00:08:07.734363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.734320 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" event={"ID":"b82b79c9-bf59-4f71-a076-a1ba614c21eb","Type":"ContainerDied","Data":"bc742722a76a27d363efdec067bab139c2b0acf92843d83f4d3c358468bdca14"} Oct 31 00:08:07.736187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.736166 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:08:07.737048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.737011 5073 generic.go:356] "Generic (PLEG): container finished" podID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerID="1fdc8d74f8a008d686bec71d38462e3d105f074d7f476d54153d2336a3475b8e" exitCode=0 Oct 31 00:08:07.737048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.737037 5073 generic.go:356] "Generic (PLEG): container finished" podID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerID="050bdb3946d0d7fb6e537be499ca7d448ae299c30b6b8497fd373b5bc193e3c3" exitCode=0 Oct 31 00:08:07.737048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.737046 5073 generic.go:356] "Generic (PLEG): container finished" podID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerID="9b0f106f670a528080e3650931fdb0601b478ba856ba6a54b5db86227a3f9d23" exitCode=0 Oct 31 00:08:07.737427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.737408 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerDied","Data":"1fdc8d74f8a008d686bec71d38462e3d105f074d7f476d54153d2336a3475b8e"} Oct 31 00:08:07.737503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.737462 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerDied","Data":"050bdb3946d0d7fb6e537be499ca7d448ae299c30b6b8497fd373b5bc193e3c3"} Oct 31 00:08:07.737503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.737486 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerDied","Data":"9b0f106f670a528080e3650931fdb0601b478ba856ba6a54b5db86227a3f9d23"} Oct 31 00:08:07.740874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.738325 5073 generic.go:356] "Generic (PLEG): container finished" podID="712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" containerID="273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f" exitCode=0 Oct 31 00:08:07.740874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.738359 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" event={"ID":"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7","Type":"ContainerDied","Data":"273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f"} Oct 31 00:08:07.740874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.738374 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" Oct 31 00:08:07.740874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.738396 5073 scope.go:122] "RemoveContainer" containerID="273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f" Oct 31 00:08:07.740874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.738383 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d" event={"ID":"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7","Type":"ContainerDied","Data":"8c4d7a3a81234108b5a828823699227f87b15aa8e8bdda62f37f1c7a97616bc5"} Oct 31 00:08:07.761259 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.761238 5073 scope.go:122] "RemoveContainer" containerID="273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f" Oct 31 00:08:07.763772 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:08:07.763516 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f\": container with ID starting with 273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f not found: ID does not exist" containerID="273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f" Oct 31 00:08:07.763772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.763554 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f"} err="failed to get container status \"273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f\": rpc error: code = NotFound desc = could not find container \"273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f\": container with ID starting with 273af7bec77664203625a0e31a14d6fc6f89e91b39e2b2406716e1aa483f140f not found: ID does not exist" Oct 31 00:08:07.793570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.793547 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5\" (UniqueName: \"kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5\") pod \"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7\" (UID: \"712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7\") " Oct 31 00:08:07.797293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.797261 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5" pod "712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" (UID: "712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7"). InnerVolumeSpecName "kube-api-access-tgpz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.894452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.894423 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgpz5\" (UniqueName: \"kubernetes.io/projected/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7-kube-api-access-tgpz5\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.901342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.901319 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:07.928531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.928501 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:08:07.995089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995060 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert\") pod \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " Oct 31 00:08:07.995089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995095 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal\") pod \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " Oct 31 00:08:07.995330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995135 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs\") pod \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " Oct 31 00:08:07.995330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995165 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz\" (UniqueName: \"kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz\") pod \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " Oct 31 00:08:07.995330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995197 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert\") pod \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " Oct 31 00:08:07.995330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995269 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs\") pod \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " Oct 31 00:08:07.995330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995306 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54\" (UniqueName: \"kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54\") pod \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " Oct 31 00:08:07.995576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995393 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\") pod \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\" (UID: \"2e93e13b-02f7-4f7c-b498-32ec8d0405e1\") " Oct 31 00:08:07.995576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995423 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal\") pod \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\" (UID: \"b82b79c9-bf59-4f71-a076-a1ba614c21eb\") " Oct 31 00:08:07.995576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995539 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal" pod "2e93e13b-02f7-4f7c-b498-32ec8d0405e1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.995895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995841 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal" pod "b82b79c9-bf59-4f71-a076-a1ba614c21eb" (UID: "b82b79c9-bf59-4f71-a076-a1ba614c21eb"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.995895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.995848 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.997707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.997676 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert" pod "2e93e13b-02f7-4f7c-b498-32ec8d0405e1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.998075 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.998045 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz" pod "b82b79c9-bf59-4f71-a076-a1ba614c21eb" (UID: "b82b79c9-bf59-4f71-a076-a1ba614c21eb"). InnerVolumeSpecName "kube-api-access-x9xcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.998075 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.998062 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54" pod "2e93e13b-02f7-4f7c-b498-32ec8d0405e1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1"). InnerVolumeSpecName "kube-api-access-9dr54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.998273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.998159 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert" pod "b82b79c9-bf59-4f71-a076-a1ba614c21eb" (UID: "b82b79c9-bf59-4f71-a076-a1ba614c21eb"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.998365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.998348 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs" pod "2e93e13b-02f7-4f7c-b498-32ec8d0405e1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.998635 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.998618 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs" pod "2e93e13b-02f7-4f7c-b498-32ec8d0405e1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.998714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:07.998634 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs" pod "2e93e13b-02f7-4f7c-b498-32ec8d0405e1" (UID: "2e93e13b-02f7-4f7c-b498-32ec8d0405e1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.060006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.059978 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d"] Oct 31 00:08:08.067078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.067053 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-bp62d"] Oct 31 00:08:08.096247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096223 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dr54\" (UniqueName: \"kubernetes.io/projected/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-kube-api-access-9dr54\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096247 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096259 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b82b79c9-bf59-4f71-a076-a1ba614c21eb-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096268 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096279 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096288 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9xcz\" (UniqueName: \"kubernetes.io/projected/b82b79c9-bf59-4f71-a076-a1ba614c21eb-kube-api-access-x9xcz\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096298 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b82b79c9-bf59-4f71-a076-a1ba614c21eb-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.096387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.096308 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e93e13b-02f7-4f7c-b498-32ec8d0405e1-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.150416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.150393 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:08:08.196852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.196828 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.196987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.196870 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.196987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.196913 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.196987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.196956 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.196983 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197031 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197133 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197164 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197212 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197247 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197275 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197306 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:08.197517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197316 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.197517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197396 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc\") pod \"efa4d324-ae53-4d1b-b3e4-12e520b92266\" (UID: \"efa4d324-ae53-4d1b-b3e4-12e520b92266\") " Oct 31 00:08:08.198059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.197994 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.198059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.198006 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:08.199767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.199725 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:08.199920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.199893 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.201055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.200897 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:08.201055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.200908 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.201055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.200954 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "kube-api-access-xblkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:08.201901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.201881 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.202261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.202232 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.202533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.202508 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.202825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.202800 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:08.207002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.206964 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:08:08.215320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.215298 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config" pod "efa4d324-ae53-4d1b-b3e4-12e520b92266" (UID: "efa4d324-ae53-4d1b-b3e4-12e520b92266"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.298403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298379 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298400 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298412 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298424 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xblkc\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-kube-api-access-xblkc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298433 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298443 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efa4d324-ae53-4d1b-b3e4-12e520b92266-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298452 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298462 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298471 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efa4d324-ae53-4d1b-b3e4-12e520b92266-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298496 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298505 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efa4d324-ae53-4d1b-b3e4-12e520b92266-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.298545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.298515 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/efa4d324-ae53-4d1b-b3e4-12e520b92266-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.313872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.313856 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:08:08.398916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.398864 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" Oct 31 00:08:08.744941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.744910 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:08:08.745108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.744918 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"efa4d324-ae53-4d1b-b3e4-12e520b92266","Type":"ContainerDied","Data":"1955ea28fa12f9aadcf772df2638024099a3bd08ebcdc3f6811a737f9e678932"} Oct 31 00:08:08.745108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.745053 5073 scope.go:122] "RemoveContainer" containerID="1fdc8d74f8a008d686bec71d38462e3d105f074d7f476d54153d2336a3475b8e" Oct 31 00:08:08.749182 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.747892 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e93e13b-02f7-4f7c-b498-32ec8d0405e1","Type":"ContainerDied","Data":"aa62770a8906bc8a4e158c2f83f3e2cd95568cf98611ff239bc95a66fea0fa02"} Oct 31 00:08:08.749182 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.747942 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:08.756749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.756726 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" event={"ID":"b82b79c9-bf59-4f71-a076-a1ba614c21eb","Type":"ContainerDied","Data":"6ac840050c0aeea6ab35947e62619e0955270d04e7a4c50a080461584cace7f4"} Oct 31 00:08:08.756877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.756754 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz" Oct 31 00:08:08.761456 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.761441 5073 scope.go:122] "RemoveContainer" containerID="050bdb3946d0d7fb6e537be499ca7d448ae299c30b6b8497fd373b5bc193e3c3" Oct 31 00:08:08.769485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.769470 5073 scope.go:122] "RemoveContainer" containerID="9b0f106f670a528080e3650931fdb0601b478ba856ba6a54b5db86227a3f9d23" Oct 31 00:08:08.777158 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.777141 5073 scope.go:122] "RemoveContainer" containerID="b0b70a80fefb63ba966b4846b80c326d00750157c4983143255e8237e3a5b911" Oct 31 00:08:08.780458 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.780438 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:08:08.784972 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.784955 5073 scope.go:122] "RemoveContainer" containerID="7d29998c0a3b4628290d4a9fd423e1c8e79a05d8d59c31e25e99d82e37ff2e38" Oct 31 00:08:08.787353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.787332 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:08:08.792876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.792857 5073 scope.go:122] "RemoveContainer" containerID="bc742722a76a27d363efdec067bab139c2b0acf92843d83f4d3c358468bdca14" Oct 31 00:08:08.801625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.801604 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:08:08.809201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.809178 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:08:08.820832 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.820812 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz"] Oct 31 00:08:08.825570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:08.825551 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7b577df47b-8v8vz"] Oct 31 00:08:10.058533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.058500 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e93e13b-02f7-4f7c-b498-32ec8d0405e1" path="/var/lib/kubelet/pods/2e93e13b-02f7-4f7c-b498-32ec8d0405e1/volumes" Oct 31 00:08:10.058920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.058812 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" path="/var/lib/kubelet/pods/712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7/volumes" Oct 31 00:08:10.059133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.059121 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82b79c9-bf59-4f71-a076-a1ba614c21eb" path="/var/lib/kubelet/pods/b82b79c9-bf59-4f71-a076-a1ba614c21eb/volumes" Oct 31 00:08:10.059776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.059762 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" path="/var/lib/kubelet/pods/efa4d324-ae53-4d1b-b3e4-12e520b92266/volumes" Oct 31 00:08:10.268079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.268045 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:08:10.269378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269327 5073 cpu_manager.go:414] "containerMap: removing container" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="init-config-reloader" Oct 31 00:08:10.269534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269514 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="init-config-reloader" Oct 31 00:08:10.269605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269551 5073 cpu_manager.go:414] "containerMap: removing container" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="prometheus" Oct 31 00:08:10.269605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269559 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="prometheus" Oct 31 00:08:10.269605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269571 5073 cpu_manager.go:414] "containerMap: removing container" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="config-reloader" Oct 31 00:08:10.269605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269579 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="config-reloader" Oct 31 00:08:10.269605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269593 5073 cpu_manager.go:414] "containerMap: removing container" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="thanos-sidecar" Oct 31 00:08:10.269605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269601 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="thanos-sidecar" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269622 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2e93e13b-02f7-4f7c-b498-32ec8d0405e1" containerName="otc-container" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269629 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e93e13b-02f7-4f7c-b498-32ec8d0405e1" containerName="otc-container" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269639 5073 cpu_manager.go:414] "containerMap: removing container" podUID="712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" containerName="thanos-querier" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269646 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" containerName="thanos-querier" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269658 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b82b79c9-bf59-4f71-a076-a1ba614c21eb" containerName="ta-container" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269665 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82b79c9-bf59-4f71-a076-a1ba614c21eb" containerName="ta-container" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269772 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e93e13b-02f7-4f7c-b498-32ec8d0405e1" containerName="otc-container" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269808 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="712a8c7b-5cfe-4138-a7c9-9b15d8ab34f7" containerName="thanos-querier" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269818 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82b79c9-bf59-4f71-a076-a1ba614c21eb" containerName="ta-container" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269834 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="prometheus" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269844 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="config-reloader" Oct 31 00:08:10.269964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.269853 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa4d324-ae53-4d1b-b3e4-12e520b92266" containerName="thanos-sidecar" Oct 31 00:08:10.275370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.275351 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.293734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.293713 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:08:10.373769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.373699 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:08:10.373769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.373762 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:08:10.373950 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.373809 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w6zng\"" Oct 31 00:08:10.378376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.378354 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:08:10.378496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.378477 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:08:10.378565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.378548 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:08:10.415586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.415565 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.415657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.415592 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prm4t\" (UniqueName: \"kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.415657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.415620 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.415743 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.415719 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.516898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.516869 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.517006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.516920 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.517006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.516992 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.517129 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.517018 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prm4t\" (UniqueName: \"kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.517687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.517590 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.519371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.519348 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.519517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.519498 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.531112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.531091 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prm4t\" (UniqueName: \"kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t\") pod \"data-science-collector-collector-1\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.596331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.596312 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:08:10.876575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.876538 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:08:10.879551 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:08:10.879528 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18c8b6a_c5a9_4897_9e5a_d5ed450ee1a9.slice/crio-72f1450c890d6c10d10ecc48ffd0a6dde98f0719ddb3aaca36aae1aa86869f36 WatchSource:0}: Error finding container 72f1450c890d6c10d10ecc48ffd0a6dde98f0719ddb3aaca36aae1aa86869f36: Status 404 returned error can't find the container with id 72f1450c890d6c10d10ecc48ffd0a6dde98f0719ddb3aaca36aae1aa86869f36 Oct 31 00:08:10.881628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.881598 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:08:10.881681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:10.881662 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:08:11.773892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:11.773861 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9","Type":"ContainerStarted","Data":"f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75"} Oct 31 00:08:11.774277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:11.773914 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9","Type":"ContainerStarted","Data":"72f1450c890d6c10d10ecc48ffd0a6dde98f0719ddb3aaca36aae1aa86869f36"} Oct 31 00:08:11.795553 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:11.795501 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7954872640000001 podStartE2EDuration="1.795487264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:11.793969389 +0000 UTC m=+575.854744085" watchObservedRunningTime="2026-10-31 00:08:11.795487264 +0000 UTC m=+575.856262020" Oct 31 00:08:13.085095 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:08:13.085064 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:08:36.107293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:36.107263 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:08:36.108867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:36.108845 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:08:36.114426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:36.114403 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:08:36.115660 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:08:36.115640 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:08:43.085330 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:08:43.085296 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:09:00.314160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.314130 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:09:00.314666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.314348 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" containerName="otc-container" containerID="cri-o://f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75" gracePeriod=30 Oct 31 00:09:00.574148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.574096 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:00.584999 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.584978 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t\" (UniqueName: \"kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t\") pod \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " Oct 31 00:09:00.585098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.585031 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs\") pod \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " Oct 31 00:09:00.585098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.585068 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal\") pod \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " Oct 31 00:09:00.585098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.585087 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs\") pod \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\" (UID: \"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9\") " Oct 31 00:09:00.585443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.585420 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal" pod "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" (UID: "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:00.587225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.587197 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t" pod "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" (UID: "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9"). InnerVolumeSpecName "kube-api-access-prm4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:00.587310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.587219 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs" pod "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" (UID: "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:00.587310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.587237 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs" pod "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" (UID: "e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:00.686413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.686388 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prm4t\" (UniqueName: \"kubernetes.io/projected/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-kube-api-access-prm4t\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.686413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.686411 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.686555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.686421 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.686555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.686431 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.987790 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.987763 5073 generic.go:356] "Generic (PLEG): container finished" podID="e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" containerID="f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75" exitCode=0 Oct 31 00:09:00.987933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.987832 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:00.987933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.987855 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9","Type":"ContainerDied","Data":"f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75"} Oct 31 00:09:00.987933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.987917 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9","Type":"ContainerDied","Data":"72f1450c890d6c10d10ecc48ffd0a6dde98f0719ddb3aaca36aae1aa86869f36"} Oct 31 00:09:00.988033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.987939 5073 scope.go:122] "RemoveContainer" containerID="f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75" Oct 31 00:09:00.997735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.997714 5073 scope.go:122] "RemoveContainer" containerID="f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75" Oct 31 00:09:00.998046 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:09:00.998023 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75\": container with ID starting with f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75 not found: ID does not exist" containerID="f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75" Oct 31 00:09:00.998169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:00.998058 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75"} err="failed to get container status \"f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75\": rpc error: code = NotFound desc = could not find container \"f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75\": container with ID starting with f039fa51b0149fd34d64bec72ea39da6465af1c174b823b347d5797933a75d75 not found: ID does not exist" Oct 31 00:09:01.011761 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:01.011737 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:09:01.017761 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:01.017741 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:09:02.058223 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:02.058184 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" path="/var/lib/kubelet/pods/e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9/volumes" Oct 31 00:09:08.790492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.790449 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx"] Oct 31 00:09:08.791002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.790984 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" containerName="otc-container" Oct 31 00:09:08.791049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.791005 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" containerName="otc-container" Oct 31 00:09:08.791131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.791118 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18c8b6a-c5a9-4897-9e5a-d5ed450ee1a9" containerName="otc-container" Oct 31 00:09:08.795377 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.795358 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:08.799893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.799867 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:09:08.800426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.800404 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:09:08.801035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.801013 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:09:08.801338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.801299 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx"] Oct 31 00:09:08.856474 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.856450 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8sd\" (UniqueName: \"kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx\" (UID: \"5a6bac2b-8485-4e32-bc18-e0373484c2d4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:08.957131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.957096 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8sd\" (UniqueName: \"kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx\" (UID: \"5a6bac2b-8485-4e32-bc18-e0373484c2d4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:08.971613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:08.971591 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8sd\" (UniqueName: \"kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx\" (UID: \"5a6bac2b-8485-4e32-bc18-e0373484c2d4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:09.107772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.107719 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:09.154915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.154881 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:09:09.159334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.159310 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.176212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.176173 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:09:09.176422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.176355 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:09:09.233663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.233634 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg"] Oct 31 00:09:09.237026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.237000 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.248637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.248584 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg"] Oct 31 00:09:09.254731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.254692 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:09:09.255105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255085 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:09:09.255105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255098 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:09:09.255277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255107 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:09:09.255277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255151 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:09:09.255277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255171 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:09:09.255277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255171 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-xcl8v\"" Oct 31 00:09:09.255696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.255669 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-fmdlr\"" Oct 31 00:09:09.259601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.259578 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzcn\" (UniqueName: \"kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.259731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.259617 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.259731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.259637 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.259731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.259660 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.259917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.259808 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.259917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.259857 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.360972 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361022 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361051 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361068 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361094 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361116 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.361244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361137 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.361244 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:09:09.361177 5073 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:09:09.361515 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:09:09.361283 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs podName:cce35ed2-1dfd-4a42-82ca-cb3741fa3478 nodeName:}" failed. No retries permitted until 2026-10-31 00:09:09.861262614 +0000 UTC m=+633.922037287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:09:09.361515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361318 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v54n4\" (UniqueName: \"kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.361515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361390 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzcn\" (UniqueName: \"kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.361678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.361525 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.363652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.363631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.363762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.363694 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.363825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.363771 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.375734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.375706 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzcn\" (UniqueName: \"kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.395850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.395826 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx"] Oct 31 00:09:09.398493 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:09:09.398473 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a6bac2b_8485_4e32_bc18_e0373484c2d4.slice/crio-de59c7c66b9192c87b532e170148a1725ff261ba4fd0715e027705a7c71d5d38 WatchSource:0}: Error finding container de59c7c66b9192c87b532e170148a1725ff261ba4fd0715e027705a7c71d5d38: Status 404 returned error can't find the container with id de59c7c66b9192c87b532e170148a1725ff261ba4fd0715e027705a7c71d5d38 Oct 31 00:09:09.462655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.462629 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.462759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.462663 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.462759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.462696 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v54n4\" (UniqueName: \"kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.463360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.463339 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.464929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.464912 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.474591 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.474571 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v54n4\" (UniqueName: \"kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4\") pod \"data-science-collector-targetallocator-7d5f95cc56-9ggwg\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.550810 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.550770 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:09.824045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.824017 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg"] Oct 31 00:09:09.827100 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:09:09.827075 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248c9918_cb59_4f14_8d61_fba74f6d15a5.slice/crio-d2b50155aa9e4b5c6b7d6e49945ceab3c41b8696010dedc0ae7b454db8d9f07c WatchSource:0}: Error finding container d2b50155aa9e4b5c6b7d6e49945ceab3c41b8696010dedc0ae7b454db8d9f07c: Status 404 returned error can't find the container with id d2b50155aa9e4b5c6b7d6e49945ceab3c41b8696010dedc0ae7b454db8d9f07c Oct 31 00:09:09.829064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.829031 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:09:09.829147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.829086 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:09:09.865942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.865920 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:09.868438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:09.868415 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:10.032009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.031975 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" event={"ID":"5a6bac2b-8485-4e32-bc18-e0373484c2d4","Type":"ContainerStarted","Data":"da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694"} Oct 31 00:09:10.032209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.032015 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" event={"ID":"5a6bac2b-8485-4e32-bc18-e0373484c2d4","Type":"ContainerStarted","Data":"de59c7c66b9192c87b532e170148a1725ff261ba4fd0715e027705a7c71d5d38"} Oct 31 00:09:10.033593 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.033568 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" event={"ID":"248c9918-cb59-4f14-8d61-fba74f6d15a5","Type":"ContainerStarted","Data":"1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f"} Oct 31 00:09:10.033716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.033599 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" event={"ID":"248c9918-cb59-4f14-8d61-fba74f6d15a5","Type":"ContainerStarted","Data":"d2b50155aa9e4b5c6b7d6e49945ceab3c41b8696010dedc0ae7b454db8d9f07c"} Oct 31 00:09:10.033776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.033740 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:10.035527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.035506 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:09:10.051725 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.051687 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" podStartSLOduration=2.051676112 podStartE2EDuration="2.051676112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:10.051068189 +0000 UTC m=+634.111842886" watchObservedRunningTime="2026-10-31 00:09:10.051676112 +0000 UTC m=+634.112450801" Oct 31 00:09:10.074240 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.074159 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" podStartSLOduration=1.074147 podStartE2EDuration="1.074147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:10.072087661 +0000 UTC m=+634.132862393" watchObservedRunningTime="2026-10-31 00:09:10.074147 +0000 UTC m=+634.134921692" Oct 31 00:09:10.084257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.084233 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:09:10.368005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.367657 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:09:10.370514 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:09:10.370483 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce35ed2_1dfd_4a42_82ca_cb3741fa3478.slice/crio-21b2bfa0f25db54cddc3212e947c71f2826f9140284b100017af3384dd8546b6 WatchSource:0}: Error finding container 21b2bfa0f25db54cddc3212e947c71f2826f9140284b100017af3384dd8546b6: Status 404 returned error can't find the container with id 21b2bfa0f25db54cddc3212e947c71f2826f9140284b100017af3384dd8546b6 Oct 31 00:09:10.372831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.372799 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:09:10.372895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:10.372863 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:09:11.039524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:11.039489 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cce35ed2-1dfd-4a42-82ca-cb3741fa3478","Type":"ContainerStarted","Data":"0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1"} Oct 31 00:09:11.039959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:11.039549 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cce35ed2-1dfd-4a42-82ca-cb3741fa3478","Type":"ContainerStarted","Data":"21b2bfa0f25db54cddc3212e947c71f2826f9140284b100017af3384dd8546b6"} Oct 31 00:09:11.060438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:11.060378 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.060360726 podStartE2EDuration="2.060360726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:11.059410141 +0000 UTC m=+635.120184838" watchObservedRunningTime="2026-10-31 00:09:11.060360726 +0000 UTC m=+635.121135424" Oct 31 00:09:13.084525 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:09:13.084491 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:09:14.262729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.262677 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm"] Oct 31 00:09:14.266816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.266778 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:09:14.274171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.274066 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm"] Oct 31 00:09:14.401213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.401183 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmlw\" (UniqueName: \"kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm\" (UID: \"16180007-37f7-4df5-88cf-3a609ee802d4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:09:14.502570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.502537 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmlw\" (UniqueName: \"kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm\" (UID: \"16180007-37f7-4df5-88cf-3a609ee802d4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:09:14.515358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.515304 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmlw\" (UniqueName: \"kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm\" (UID: \"16180007-37f7-4df5-88cf-3a609ee802d4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:09:14.580228 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.580190 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:09:14.860889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:14.860810 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm"] Oct 31 00:09:14.863634 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:09:14.863606 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16180007_37f7_4df5_88cf_3a609ee802d4.slice/crio-3408179f57632c37a6a6d9ac27ff23fe6cf52e80231dd20bd4d4bbb218d8a558 WatchSource:0}: Error finding container 3408179f57632c37a6a6d9ac27ff23fe6cf52e80231dd20bd4d4bbb218d8a558: Status 404 returned error can't find the container with id 3408179f57632c37a6a6d9ac27ff23fe6cf52e80231dd20bd4d4bbb218d8a558 Oct 31 00:09:15.057648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.057607 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" event={"ID":"16180007-37f7-4df5-88cf-3a609ee802d4","Type":"ContainerStarted","Data":"932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc"} Oct 31 00:09:15.057893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.057659 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" event={"ID":"16180007-37f7-4df5-88cf-3a609ee802d4","Type":"ContainerStarted","Data":"3408179f57632c37a6a6d9ac27ff23fe6cf52e80231dd20bd4d4bbb218d8a558"} Oct 31 00:09:15.079162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.079119 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" podStartSLOduration=1.079105153 podStartE2EDuration="1.079105153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:15.077869579 +0000 UTC m=+639.138644276" watchObservedRunningTime="2026-10-31 00:09:15.079105153 +0000 UTC m=+639.139879922" Oct 31 00:09:15.109528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.109501 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx"] Oct 31 00:09:15.109672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.109654 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" podUID="5a6bac2b-8485-4e32-bc18-e0373484c2d4" containerName="thanos-querier" containerID="cri-o://da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694" gracePeriod=30 Oct 31 00:09:15.226418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.226392 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:09:15.230328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.230308 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.238387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.238284 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:09:15.238387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.238306 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:09:15.238387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.238381 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:09:15.247478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.247447 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:09:15.257935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.257900 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:09:15.257935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.257900 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:09:15.258096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.257930 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:09:15.258096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.257928 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:09:15.258326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.258302 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:09:15.258452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.258371 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:09:15.258452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.258441 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-46rq9\"" Oct 31 00:09:15.309995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.309958 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.309995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.309990 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310010 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310097 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310129 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310180 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310209 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310249 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310308 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.310486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310445 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd86l\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.311013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310501 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.311013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.310532 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.351108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.351088 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:15.410872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.410813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd\" (UniqueName: \"kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd\") pod \"5a6bac2b-8485-4e32-bc18-e0373484c2d4\" (UID: \"5a6bac2b-8485-4e32-bc18-e0373484c2d4\") " Oct 31 00:09:15.410980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.410959 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.410983 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411005 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411109 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411157 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411225 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd86l\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411273 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411312 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411353 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411383 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411514 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411417 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.411514 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411450 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.412613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.411872 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.412613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.412322 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.412613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.412332 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.413725 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.413682 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd" pod "5a6bac2b-8485-4e32-bc18-e0373484c2d4" (UID: "5a6bac2b-8485-4e32-bc18-e0373484c2d4"). InnerVolumeSpecName "kube-api-access-tm8sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:15.414169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.414132 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.414293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.414270 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.414366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.414342 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.414545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.414524 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.414632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.414618 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.415233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.415211 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.415961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.415942 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.416199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.416180 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.424888 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.424868 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd86l\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:15.512277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:15.512256 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm8sd\" (UniqueName: \"kubernetes.io/projected/5a6bac2b-8485-4e32-bc18-e0373484c2d4-kube-api-access-tm8sd\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:16.063882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.063842 5073 generic.go:356] "Generic (PLEG): container finished" podID="5a6bac2b-8485-4e32-bc18-e0373484c2d4" containerID="da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694" exitCode=0 Oct 31 00:09:16.063882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.063875 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" Oct 31 00:09:16.064112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.063917 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" event={"ID":"5a6bac2b-8485-4e32-bc18-e0373484c2d4","Type":"ContainerDied","Data":"da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694"} Oct 31 00:09:16.064112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.063960 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx" event={"ID":"5a6bac2b-8485-4e32-bc18-e0373484c2d4","Type":"ContainerDied","Data":"de59c7c66b9192c87b532e170148a1725ff261ba4fd0715e027705a7c71d5d38"} Oct 31 00:09:16.064112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.063980 5073 scope.go:122] "RemoveContainer" containerID="da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694" Oct 31 00:09:16.073573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.073550 5073 scope.go:122] "RemoveContainer" containerID="da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694" Oct 31 00:09:16.073813 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:09:16.073772 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694\": container with ID starting with da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694 not found: ID does not exist" containerID="da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694" Oct 31 00:09:16.073868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.073822 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694"} err="failed to get container status \"da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694\": rpc error: code = NotFound desc = could not find container \"da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694\": container with ID starting with da974f937ffd220107b950be6e3a893ccb587767fb3d8b1a73536b0bf0d25694 not found: ID does not exist" Oct 31 00:09:16.082906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.082880 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx"] Oct 31 00:09:16.091302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:16.091283 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-pqxvx"] Oct 31 00:09:18.058863 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:18.058825 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6bac2b-8485-4e32-bc18-e0373484c2d4" path="/var/lib/kubelet/pods/5a6bac2b-8485-4e32-bc18-e0373484c2d4/volumes" Oct 31 00:09:18.842146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:18.842110 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:18.848953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:18.848925 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:18.942920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:18.942885 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:18.943052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:18.943010 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:18.946301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:18.946284 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:19.037308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:19.037283 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:19.052930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:19.052909 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:19.152151 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:19.152095 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:19.440094 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:19.440015 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:09:19.443365 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:09:19.443330 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc47611f_8016_48e5_b277_8e3f5b4068f8.slice/crio-b496b489a5ae481fcbef190c9df96fb1822de08ee96f5da7973e00552d7576be WatchSource:0}: Error finding container b496b489a5ae481fcbef190c9df96fb1822de08ee96f5da7973e00552d7576be: Status 404 returned error can't find the container with id b496b489a5ae481fcbef190c9df96fb1822de08ee96f5da7973e00552d7576be Oct 31 00:09:20.082952 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:20.082881 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerStarted","Data":"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6"} Oct 31 00:09:20.083074 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:20.082958 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerStarted","Data":"b496b489a5ae481fcbef190c9df96fb1822de08ee96f5da7973e00552d7576be"} Oct 31 00:09:24.101851 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:24.101807 5073 generic.go:356] "Generic (PLEG): container finished" podID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerID="afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6" exitCode=0 Oct 31 00:09:24.102265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:24.101859 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerDied","Data":"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6"} Oct 31 00:09:25.109717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:25.109687 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerStarted","Data":"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4"} Oct 31 00:09:25.110069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:25.109748 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerStarted","Data":"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4"} Oct 31 00:09:25.110069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:25.109775 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerStarted","Data":"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f"} Oct 31 00:09:25.144197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:25.144099 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=6.659535154 podStartE2EDuration="10.14407994s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484544786s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:25.141984723 +0000 UTC m=+649.202759409" watchObservedRunningTime="2026-10-31 00:09:25.14407994 +0000 UTC m=+649.204854639" Oct 31 00:09:29.152545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:29.152513 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:34.152604 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:34.152573 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:34.158042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:34.158019 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:35.158418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:09:35.158392 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:09:43.085640 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:09:43.085605 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:10:01.483021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.482706 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm"] Oct 31 00:10:01.484115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.484065 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" podUID="16180007-37f7-4df5-88cf-3a609ee802d4" containerName="thanos-querier" containerID="cri-o://932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc" gracePeriod=30 Oct 31 00:10:01.562767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.562735 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:10:01.563119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.563093 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="prometheus" containerID="cri-o://46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" gracePeriod=600 Oct 31 00:10:01.563279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.563258 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="thanos-sidecar" containerID="cri-o://916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" gracePeriod=600 Oct 31 00:10:01.563366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.563344 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="config-reloader" containerID="cri-o://e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" gracePeriod=600 Oct 31 00:10:01.772266 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.772091 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:01.772509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.772356 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="cce35ed2-1dfd-4a42-82ca-cb3741fa3478" containerName="otc-container" containerID="cri-o://0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1" gracePeriod=30 Oct 31 00:10:01.804326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.802862 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg"] Oct 31 00:10:01.804326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.803071 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" podUID="248c9918-cb59-4f14-8d61-fba74f6d15a5" containerName="ta-container" containerID="cri-o://1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f" gracePeriod=30 Oct 31 00:10:01.973608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.973208 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:10:01.976046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:01.975950 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:02.076263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076231 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076279 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076333 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076366 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076401 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076438 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076546 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076581 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076672 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076721 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw\" (UniqueName: \"kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw\") pod \"16180007-37f7-4df5-88cf-3a609ee802d4\" (UID: \"16180007-37f7-4df5-88cf-3a609ee802d4\") " Oct 31 00:10:02.076938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076750 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.076938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076919 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.077093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076953 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.077093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.076985 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file\") pod \"fc47611f-8016-48e5-b277-8e3f5b4068f8\" (UID: \"fc47611f-8016-48e5-b277-8e3f5b4068f8\") " Oct 31 00:10:02.077609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.077579 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.077772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.077740 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.079447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.079395 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.083318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.083267 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.083479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.083451 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:02.083697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.083668 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.085841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.085808 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.086358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.086324 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.087508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.087472 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "kube-api-access-pd86l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.088046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.087931 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.089372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.089347 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw" pod "16180007-37f7-4df5-88cf-3a609ee802d4" (UID: "16180007-37f7-4df5-88cf-3a609ee802d4"). InnerVolumeSpecName "kube-api-access-2lmlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.097369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.097227 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.097693 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.097668 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:10:02.117663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.117638 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config" pod "fc47611f-8016-48e5-b277-8e3f5b4068f8" (UID: "fc47611f-8016-48e5-b277-8e3f5b4068f8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.130814 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.130777 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:02.148067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.148047 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:10:02.178367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178340 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs\") pod \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " Oct 31 00:10:02.178508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178380 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal\") pod \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " Oct 31 00:10:02.178508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178424 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs\") pod \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " Oct 31 00:10:02.178508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178458 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\") pod \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " Oct 31 00:10:02.178673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178504 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert\") pod \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " Oct 31 00:10:02.178673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn\" (UniqueName: \"kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn\") pod \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\" (UID: \"cce35ed2-1dfd-4a42-82ca-cb3741fa3478\") " Oct 31 00:10:02.178673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178604 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert\") pod \"248c9918-cb59-4f14-8d61-fba74f6d15a5\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " Oct 31 00:10:02.178673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178640 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4\" (UniqueName: \"kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4\") pod \"248c9918-cb59-4f14-8d61-fba74f6d15a5\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " Oct 31 00:10:02.178898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.178674 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal\" (UniqueName: \"kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal\") pod \"248c9918-cb59-4f14-8d61-fba74f6d15a5\" (UID: \"248c9918-cb59-4f14-8d61-fba74f6d15a5\") " Oct 31 00:10:02.180252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.180231 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.180453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.180436 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.180678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.180661 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:10:02.180830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.180814 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmlw\" (UniqueName: \"kubernetes.io/projected/16180007-37f7-4df5-88cf-3a609ee802d4-kube-api-access-2lmlw\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.180936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.180922 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.181035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181022 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.181125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181112 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.181213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181198 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181293 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181315 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd86l\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-kube-api-access-pd86l\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181329 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc47611f-8016-48e5-b277-8e3f5b4068f8-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181345 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc47611f-8016-48e5-b277-8e3f5b4068f8-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181361 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fc47611f-8016-48e5-b277-8e3f5b4068f8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181377 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc47611f-8016-48e5-b277-8e3f5b4068f8-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.180675 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal" pod "248c9918-cb59-4f14-8d61-fba74f6d15a5" (UID: "248c9918-cb59-4f14-8d61-fba74f6d15a5"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.182981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.181303 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal" pod "cce35ed2-1dfd-4a42-82ca-cb3741fa3478" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.183461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.183400 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn" pod "cce35ed2-1dfd-4a42-82ca-cb3741fa3478" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478"). InnerVolumeSpecName "kube-api-access-9mzcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.183797 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.183732 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs" pod "cce35ed2-1dfd-4a42-82ca-cb3741fa3478" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.188209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.188170 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4" pod "248c9918-cb59-4f14-8d61-fba74f6d15a5" (UID: "248c9918-cb59-4f14-8d61-fba74f6d15a5"). InnerVolumeSpecName "kube-api-access-v54n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.189165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.189145 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert" pod "cce35ed2-1dfd-4a42-82ca-cb3741fa3478" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.191061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.191034 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert" pod "248c9918-cb59-4f14-8d61-fba74f6d15a5" (UID: "248c9918-cb59-4f14-8d61-fba74f6d15a5"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.191203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.191081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs" pod "cce35ed2-1dfd-4a42-82ca-cb3741fa3478" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.191277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.191202 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs" pod "cce35ed2-1dfd-4a42-82ca-cb3741fa3478" (UID: "cce35ed2-1dfd-4a42-82ca-cb3741fa3478"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.196418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.196401 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:10:02.280152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.280087 5073 generic.go:356] "Generic (PLEG): container finished" podID="16180007-37f7-4df5-88cf-3a609ee802d4" containerID="932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc" exitCode=0 Oct 31 00:10:02.280152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.280129 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" Oct 31 00:10:02.280313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.280176 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" event={"ID":"16180007-37f7-4df5-88cf-3a609ee802d4","Type":"ContainerDied","Data":"932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc"} Oct 31 00:10:02.280313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.280213 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm" event={"ID":"16180007-37f7-4df5-88cf-3a609ee802d4","Type":"ContainerDied","Data":"3408179f57632c37a6a6d9ac27ff23fe6cf52e80231dd20bd4d4bbb218d8a558"} Oct 31 00:10:02.280313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.280231 5073 scope.go:122] "RemoveContainer" containerID="932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc" Oct 31 00:10:02.281926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.281903 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzcn\" (UniqueName: \"kubernetes.io/projected/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-kube-api-access-9mzcn\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.281933 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.281952 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/248c9918-cb59-4f14-8d61-fba74f6d15a5-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.281967 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v54n4\" (UniqueName: \"kubernetes.io/projected/248c9918-cb59-4f14-8d61-fba74f6d15a5-kube-api-access-v54n4\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.281982 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/248c9918-cb59-4f14-8d61-fba74f6d15a5-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.281996 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282010 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282024 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282038 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282351 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282053 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cce35ed2-1dfd-4a42-82ca-cb3741fa3478-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.282945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282925 5073 generic.go:356] "Generic (PLEG): container finished" podID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerID="916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" exitCode=0 Oct 31 00:10:02.283065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282944 5073 generic.go:356] "Generic (PLEG): container finished" podID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerID="e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" exitCode=0 Oct 31 00:10:02.283065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.282953 5073 generic.go:356] "Generic (PLEG): container finished" podID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerID="46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" exitCode=0 Oct 31 00:10:02.283065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.283013 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerDied","Data":"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4"} Oct 31 00:10:02.283065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.283024 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:02.283253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.283072 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerDied","Data":"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4"} Oct 31 00:10:02.283253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.283103 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerDied","Data":"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f"} Oct 31 00:10:02.283253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.283139 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"fc47611f-8016-48e5-b277-8e3f5b4068f8","Type":"ContainerDied","Data":"b496b489a5ae481fcbef190c9df96fb1822de08ee96f5da7973e00552d7576be"} Oct 31 00:10:02.284683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.284664 5073 generic.go:356] "Generic (PLEG): container finished" podID="248c9918-cb59-4f14-8d61-fba74f6d15a5" containerID="1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f" exitCode=0 Oct 31 00:10:02.284812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.284718 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" Oct 31 00:10:02.284812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.284750 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" event={"ID":"248c9918-cb59-4f14-8d61-fba74f6d15a5","Type":"ContainerDied","Data":"1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f"} Oct 31 00:10:02.284928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.284821 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg" event={"ID":"248c9918-cb59-4f14-8d61-fba74f6d15a5","Type":"ContainerDied","Data":"d2b50155aa9e4b5c6b7d6e49945ceab3c41b8696010dedc0ae7b454db8d9f07c"} Oct 31 00:10:02.286329 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.286307 5073 generic.go:356] "Generic (PLEG): container finished" podID="cce35ed2-1dfd-4a42-82ca-cb3741fa3478" containerID="0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1" exitCode=0 Oct 31 00:10:02.286429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.286350 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cce35ed2-1dfd-4a42-82ca-cb3741fa3478","Type":"ContainerDied","Data":"0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1"} Oct 31 00:10:02.286429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.286381 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cce35ed2-1dfd-4a42-82ca-cb3741fa3478","Type":"ContainerDied","Data":"21b2bfa0f25db54cddc3212e947c71f2826f9140284b100017af3384dd8546b6"} Oct 31 00:10:02.286429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.286351 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:02.290650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.290627 5073 scope.go:122] "RemoveContainer" containerID="932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc" Oct 31 00:10:02.290950 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.290928 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc\": container with ID starting with 932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc not found: ID does not exist" containerID="932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc" Oct 31 00:10:02.291039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.290961 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc"} err="failed to get container status \"932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc\": rpc error: code = NotFound desc = could not find container \"932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc\": container with ID starting with 932c820845598566f6a2d231550cdf1048b2d26179371233f58bcc5438a800bc not found: ID does not exist" Oct 31 00:10:02.291039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.290983 5073 scope.go:122] "RemoveContainer" containerID="916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" Oct 31 00:10:02.299952 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.299936 5073 scope.go:122] "RemoveContainer" containerID="e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" Oct 31 00:10:02.308333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.308318 5073 scope.go:122] "RemoveContainer" containerID="46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" Oct 31 00:10:02.309237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.309153 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm"] Oct 31 00:10:02.315018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.314992 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-vlklm"] Oct 31 00:10:02.322627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.322609 5073 scope.go:122] "RemoveContainer" containerID="afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6" Oct 31 00:10:02.330205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.330166 5073 scope.go:122] "RemoveContainer" containerID="916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" Oct 31 00:10:02.330744 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.330706 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": container with ID starting with 916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4 not found: ID does not exist" containerID="916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" Oct 31 00:10:02.330860 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.330761 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4"} err="failed to get container status \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": rpc error: code = NotFound desc = could not find container \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": container with ID starting with 916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4 not found: ID does not exist" Oct 31 00:10:02.330860 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.330810 5073 scope.go:122] "RemoveContainer" containerID="e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" Oct 31 00:10:02.331137 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.331117 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": container with ID starting with e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4 not found: ID does not exist" containerID="e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" Oct 31 00:10:02.331231 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331141 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4"} err="failed to get container status \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": rpc error: code = NotFound desc = could not find container \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": container with ID starting with e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4 not found: ID does not exist" Oct 31 00:10:02.331231 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331157 5073 scope.go:122] "RemoveContainer" containerID="46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" Oct 31 00:10:02.331375 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.331351 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": container with ID starting with 46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f not found: ID does not exist" containerID="46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" Oct 31 00:10:02.331420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331384 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f"} err="failed to get container status \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": rpc error: code = NotFound desc = could not find container \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": container with ID starting with 46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f not found: ID does not exist" Oct 31 00:10:02.331420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331406 5073 scope.go:122] "RemoveContainer" containerID="afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6" Oct 31 00:10:02.331621 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.331603 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": container with ID starting with afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6 not found: ID does not exist" containerID="afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6" Oct 31 00:10:02.331679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331627 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6"} err="failed to get container status \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": rpc error: code = NotFound desc = could not find container \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": container with ID starting with afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6 not found: ID does not exist" Oct 31 00:10:02.331679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331644 5073 scope.go:122] "RemoveContainer" containerID="916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" Oct 31 00:10:02.331863 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331843 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4"} err="failed to get container status \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": rpc error: code = NotFound desc = could not find container \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": container with ID starting with 916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4 not found: ID does not exist" Oct 31 00:10:02.331910 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.331865 5073 scope.go:122] "RemoveContainer" containerID="e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" Oct 31 00:10:02.332080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332061 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4"} err="failed to get container status \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": rpc error: code = NotFound desc = could not find container \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": container with ID starting with e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4 not found: ID does not exist" Oct 31 00:10:02.332125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332081 5073 scope.go:122] "RemoveContainer" containerID="46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" Oct 31 00:10:02.332260 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332241 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f"} err="failed to get container status \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": rpc error: code = NotFound desc = could not find container \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": container with ID starting with 46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f not found: ID does not exist" Oct 31 00:10:02.332308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332261 5073 scope.go:122] "RemoveContainer" containerID="afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6" Oct 31 00:10:02.332622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332603 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6"} err="failed to get container status \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": rpc error: code = NotFound desc = could not find container \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": container with ID starting with afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6 not found: ID does not exist" Oct 31 00:10:02.332663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332622 5073 scope.go:122] "RemoveContainer" containerID="916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4" Oct 31 00:10:02.332837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332821 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4"} err="failed to get container status \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": rpc error: code = NotFound desc = could not find container \"916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4\": container with ID starting with 916f2eca4e23251c1a4b50a6ca171fa6e2e031645991fb2e3a590f21dd5bb0c4 not found: ID does not exist" Oct 31 00:10:02.332893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332837 5073 scope.go:122] "RemoveContainer" containerID="e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4" Oct 31 00:10:02.332893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.332866 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:10:02.333036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.333014 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4"} err="failed to get container status \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": rpc error: code = NotFound desc = could not find container \"e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4\": container with ID starting with e66ffc33e47e47cc525e1dab84e8d88f32985702d4e697ce5dcdb0ab31f2f8e4 not found: ID does not exist" Oct 31 00:10:02.333036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.333036 5073 scope.go:122] "RemoveContainer" containerID="46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f" Oct 31 00:10:02.333243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.333225 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f"} err="failed to get container status \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": rpc error: code = NotFound desc = could not find container \"46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f\": container with ID starting with 46ed9e8e45273b829e9b4408d548602b061a9237e8e5d698b0aeaf7a633a152f not found: ID does not exist" Oct 31 00:10:02.333287 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.333245 5073 scope.go:122] "RemoveContainer" containerID="afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6" Oct 31 00:10:02.333449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.333433 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6"} err="failed to get container status \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": rpc error: code = NotFound desc = could not find container \"afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6\": container with ID starting with afe04add54092c377ddddd9a90de4c56bd9cd88d004d8249ae1efd48ea4835e6 not found: ID does not exist" Oct 31 00:10:02.333494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.333449 5073 scope.go:122] "RemoveContainer" containerID="1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f" Oct 31 00:10:02.337462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.337443 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:10:02.340697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.340682 5073 scope.go:122] "RemoveContainer" containerID="1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f" Oct 31 00:10:02.340936 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.340920 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f\": container with ID starting with 1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f not found: ID does not exist" containerID="1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f" Oct 31 00:10:02.340998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.340939 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f"} err="failed to get container status \"1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f\": rpc error: code = NotFound desc = could not find container \"1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f\": container with ID starting with 1f5ed2b943daa8774d6162cbb9100016ef874ab9e4405a82a1c37f3e6e57788f not found: ID does not exist" Oct 31 00:10:02.340998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.340951 5073 scope.go:122] "RemoveContainer" containerID="0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1" Oct 31 00:10:02.348127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.348109 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:02.348536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.348524 5073 scope.go:122] "RemoveContainer" containerID="0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1" Oct 31 00:10:02.348778 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:02.348758 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1\": container with ID starting with 0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1 not found: ID does not exist" containerID="0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1" Oct 31 00:10:02.348849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.348799 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1"} err="failed to get container status \"0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1\": rpc error: code = NotFound desc = could not find container \"0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1\": container with ID starting with 0eabccc09d1010b0b4bb8c07b5409ab7045793312b97af6f765ead6e1b0907d1 not found: ID does not exist" Oct 31 00:10:02.352211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.352191 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:02.368947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.368925 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg"] Oct 31 00:10:02.371552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:02.371533 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7d5f95cc56-9ggwg"] Oct 31 00:10:04.059173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:04.059144 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16180007-37f7-4df5-88cf-3a609ee802d4" path="/var/lib/kubelet/pods/16180007-37f7-4df5-88cf-3a609ee802d4/volumes" Oct 31 00:10:04.059552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:04.059490 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248c9918-cb59-4f14-8d61-fba74f6d15a5" path="/var/lib/kubelet/pods/248c9918-cb59-4f14-8d61-fba74f6d15a5/volumes" Oct 31 00:10:04.059850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:04.059836 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce35ed2-1dfd-4a42-82ca-cb3741fa3478" path="/var/lib/kubelet/pods/cce35ed2-1dfd-4a42-82ca-cb3741fa3478/volumes" Oct 31 00:10:04.060496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:04.060484 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" path="/var/lib/kubelet/pods/fc47611f-8016-48e5-b277-8e3f5b4068f8/volumes" Oct 31 00:10:09.808908 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.808870 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr"] Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809180 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="thanos-sidecar" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809191 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="thanos-sidecar" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809208 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="init-config-reloader" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809213 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="init-config-reloader" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809219 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="prometheus" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809225 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="prometheus" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809236 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="config-reloader" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809241 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="config-reloader" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809254 5073 cpu_manager.go:414] "containerMap: removing container" podUID="248c9918-cb59-4f14-8d61-fba74f6d15a5" containerName="ta-container" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809258 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248c9918-cb59-4f14-8d61-fba74f6d15a5" containerName="ta-container" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809264 5073 cpu_manager.go:414] "containerMap: removing container" podUID="16180007-37f7-4df5-88cf-3a609ee802d4" containerName="thanos-querier" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809268 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16180007-37f7-4df5-88cf-3a609ee802d4" containerName="thanos-querier" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809277 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5a6bac2b-8485-4e32-bc18-e0373484c2d4" containerName="thanos-querier" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809282 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6bac2b-8485-4e32-bc18-e0373484c2d4" containerName="thanos-querier" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809288 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cce35ed2-1dfd-4a42-82ca-cb3741fa3478" containerName="otc-container" Oct 31 00:10:09.809295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809292 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce35ed2-1dfd-4a42-82ca-cb3741fa3478" containerName="otc-container" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809342 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="prometheus" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809349 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="config-reloader" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809356 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="248c9918-cb59-4f14-8d61-fba74f6d15a5" containerName="ta-container" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809361 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="16180007-37f7-4df5-88cf-3a609ee802d4" containerName="thanos-querier" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809367 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6bac2b-8485-4e32-bc18-e0373484c2d4" containerName="thanos-querier" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809373 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce35ed2-1dfd-4a42-82ca-cb3741fa3478" containerName="otc-container" Oct 31 00:10:09.809845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.809378 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc47611f-8016-48e5-b277-8e3f5b4068f8" containerName="thanos-sidecar" Oct 31 00:10:09.814326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.814304 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:09.819166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.819140 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:10:09.819320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.819303 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:10:09.819484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.819464 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:10:09.821245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.821222 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr"] Oct 31 00:10:09.940931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:09.940906 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrt25\" (UniqueName: \"kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr\" (UID: \"ae7e4e17-da4a-4f14-8397-f787cf5d1f11\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:10.041466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.041438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrt25\" (UniqueName: \"kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr\" (UID: \"ae7e4e17-da4a-4f14-8397-f787cf5d1f11\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:10.057383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.057361 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrt25\" (UniqueName: \"kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr\" (UID: \"ae7e4e17-da4a-4f14-8397-f787cf5d1f11\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:10.107656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.107586 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:10.111626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.111604 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.125251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.125229 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:10.128043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.128019 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:10.135316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.135182 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:10:10.187944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.187253 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc"] Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.196069 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.199492 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.199561 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.199644 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.199734 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.199856 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:10:10.200144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.200079 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc"] Oct 31 00:10:10.243532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243505 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqbs\" (UniqueName: \"kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.243670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243557 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.243670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243627 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.243746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243696 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.243746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243735 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.243863 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243756 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.243863 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243849 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.243951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243874 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.243951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.243896 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxpq\" (UniqueName: \"kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.266392 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.266351 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-zpbg7\"" Oct 31 00:10:10.266534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.266455 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:10:10.299263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.299247 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:10:10.344794 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344757 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.344906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344810 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.344906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344831 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.344906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344860 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.344906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344880 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxpq\" (UniqueName: \"kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.344906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344896 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqbs\" (UniqueName: \"kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.345183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344931 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.345183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.344954 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.345183 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:10.345053 5073 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:10:10.345183 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:10.345106 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs podName:3ee9b82c-2498-4b77-8a6d-f08c6c0299e7 nodeName:}" failed. No retries permitted until 2026-10-31 00:10:10.845091947 +0000 UTC m=+694.905866633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:10:10.345183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.345155 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.345548 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.345530 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.345607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.345590 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.347652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.347627 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.347746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.347712 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.348072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.348052 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.348146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.348055 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.357227 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.357205 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxpq\" (UniqueName: \"kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq\") pod \"data-science-collector-targetallocator-859d7c9d9d-svsfc\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.357654 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.357633 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqbs\" (UniqueName: \"kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.408326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.408268 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr"] Oct 31 00:10:10.411468 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:10.411437 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae7e4e17_da4a_4f14_8397_f787cf5d1f11.slice/crio-046700a068dfc85a26308e75e1c773f754c471c705d8e65cf56d0f4e2d3ded43 WatchSource:0}: Error finding container 046700a068dfc85a26308e75e1c773f754c471c705d8e65cf56d0f4e2d3ded43: Status 404 returned error can't find the container with id 046700a068dfc85a26308e75e1c773f754c471c705d8e65cf56d0f4e2d3ded43 Oct 31 00:10:10.514422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.514394 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:10.788574 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.788545 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc"] Oct 31 00:10:10.791355 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:10.791322 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode00a0ce9_8ed0_403a_9946_7aa0973b480b.slice/crio-b6157b9fecc96e0684e65baa08ac83ba368ef91dafb909edf64465dcad8d1b22 WatchSource:0}: Error finding container b6157b9fecc96e0684e65baa08ac83ba368ef91dafb909edf64465dcad8d1b22: Status 404 returned error can't find the container with id b6157b9fecc96e0684e65baa08ac83ba368ef91dafb909edf64465dcad8d1b22 Oct 31 00:10:10.793726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.793694 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:10.793868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.793748 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:10.849990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.849964 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:10.852337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:10.852311 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:11.028529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.028490 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:11.306124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.306053 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:11.309033 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:11.309000 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee9b82c_2498_4b77_8a6d_f08c6c0299e7.slice/crio-41964f83b2bc8d3f01aa482574a5edf5659235a73e27bbd1118b90d9725f739d WatchSource:0}: Error finding container 41964f83b2bc8d3f01aa482574a5edf5659235a73e27bbd1118b90d9725f739d: Status 404 returned error can't find the container with id 41964f83b2bc8d3f01aa482574a5edf5659235a73e27bbd1118b90d9725f739d Oct 31 00:10:11.311140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.311109 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:11.311221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.311166 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:11.331502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.331480 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7","Type":"ContainerStarted","Data":"41964f83b2bc8d3f01aa482574a5edf5659235a73e27bbd1118b90d9725f739d"} Oct 31 00:10:11.332907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.332880 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" event={"ID":"e00a0ce9-8ed0-403a-9946-7aa0973b480b","Type":"ContainerStarted","Data":"a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f"} Oct 31 00:10:11.333007 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.332924 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" event={"ID":"e00a0ce9-8ed0-403a-9946-7aa0973b480b","Type":"ContainerStarted","Data":"b6157b9fecc96e0684e65baa08ac83ba368ef91dafb909edf64465dcad8d1b22"} Oct 31 00:10:11.333071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.333030 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:11.334846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.334822 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" event={"ID":"ae7e4e17-da4a-4f14-8397-f787cf5d1f11","Type":"ContainerStarted","Data":"4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd"} Oct 31 00:10:11.334920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.334855 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" event={"ID":"ae7e4e17-da4a-4f14-8397-f787cf5d1f11","Type":"ContainerStarted","Data":"046700a068dfc85a26308e75e1c773f754c471c705d8e65cf56d0f4e2d3ded43"} Oct 31 00:10:11.335163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.335152 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:11.355057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.355021 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" podStartSLOduration=1.355009653 podStartE2EDuration="1.355009653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.353161891 +0000 UTC m=+695.413936580" watchObservedRunningTime="2026-10-31 00:10:11.355009653 +0000 UTC m=+695.415784349" Oct 31 00:10:11.391472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:11.391409 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" podStartSLOduration=2.391391402 podStartE2EDuration="2.391391402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.390198872 +0000 UTC m=+695.450973568" watchObservedRunningTime="2026-10-31 00:10:11.391391402 +0000 UTC m=+695.452166098" Oct 31 00:10:12.341840 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:12.341795 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7","Type":"ContainerStarted","Data":"4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926"} Oct 31 00:10:12.367702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:12.367651 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.3676338230000002 podStartE2EDuration="2.367633823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:12.366430129 +0000 UTC m=+696.427204826" watchObservedRunningTime="2026-10-31 00:10:12.367633823 +0000 UTC m=+696.428408518" Oct 31 00:10:13.085068 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:13.085038 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:10:15.433347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.433312 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db"] Oct 31 00:10:15.436918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.436897 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:10:15.444760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.444732 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db"] Oct 31 00:10:15.488819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.488772 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngw5f\" (UniqueName: \"kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db\" (UID: \"6ab588f8-26c2-4321-b967-174cdf088178\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:10:15.589475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.589445 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw5f\" (UniqueName: \"kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db\" (UID: \"6ab588f8-26c2-4321-b967-174cdf088178\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:10:15.602026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.602001 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw5f\" (UniqueName: \"kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db\" (UID: \"6ab588f8-26c2-4321-b967-174cdf088178\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:10:15.749497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:15.749470 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:10:16.032546 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.032475 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db"] Oct 31 00:10:16.035715 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:16.035673 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ab588f8_26c2_4321_b967_174cdf088178.slice/crio-40c5e0d3453b29082f5dd433323eb0d556fbb5ce2c095379a30f7a406770cfed WatchSource:0}: Error finding container 40c5e0d3453b29082f5dd433323eb0d556fbb5ce2c095379a30f7a406770cfed: Status 404 returned error can't find the container with id 40c5e0d3453b29082f5dd433323eb0d556fbb5ce2c095379a30f7a406770cfed Oct 31 00:10:16.152325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.152290 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:10:16.156199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.156178 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.174822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.174775 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:10:16.234684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234607 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:10:16.234684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234658 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:10:16.234684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234712 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:10:16.235029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234734 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:10:16.235029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234822 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:10:16.235029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234900 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:10:16.235029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.234995 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-n9gt9\"" Oct 31 00:10:16.237376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.237362 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:10:16.237497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.237416 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:10:16.237579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.237568 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:10:16.295581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295525 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295561 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295582 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295601 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295663 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295703 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295829 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295863 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295885 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.295956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295915 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.296076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.295980 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjzp\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.296076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.296022 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.360589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.360560 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" event={"ID":"6ab588f8-26c2-4321-b967-174cdf088178","Type":"ContainerStarted","Data":"1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2"} Oct 31 00:10:16.360707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.360600 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" event={"ID":"6ab588f8-26c2-4321-b967-174cdf088178","Type":"ContainerStarted","Data":"40c5e0d3453b29082f5dd433323eb0d556fbb5ce2c095379a30f7a406770cfed"} Oct 31 00:10:16.380255 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.380201 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" podStartSLOduration=1.380186439 podStartE2EDuration="1.380186439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:16.380099203 +0000 UTC m=+700.440873901" watchObservedRunningTime="2026-10-31 00:10:16.380186439 +0000 UTC m=+700.440961133" Oct 31 00:10:16.397492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397464 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.397631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397518 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.397631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397545 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.397631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397576 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjzp\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.397631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397623 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397651 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397730 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397761 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397814 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397838 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.398489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.397889 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.400262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.399932 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.400262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.400018 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.401376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.400929 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.401376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.401034 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.401376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.401337 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.402154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.402120 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.403020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.402937 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.403254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.403213 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.403804 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.403750 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.404251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.404228 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.405052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.405034 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.408045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.408024 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr"] Oct 31 00:10:16.408268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.408242 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" podUID="ae7e4e17-da4a-4f14-8397-f787cf5d1f11" containerName="thanos-querier" containerID="cri-o://4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd" gracePeriod=30 Oct 31 00:10:16.415384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.415358 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjzp\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:16.654522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.654494 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:16.699945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.699915 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25\" (UniqueName: \"kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25\") pod \"ae7e4e17-da4a-4f14-8397-f787cf5d1f11\" (UID: \"ae7e4e17-da4a-4f14-8397-f787cf5d1f11\") " Oct 31 00:10:16.702021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.701993 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25" pod "ae7e4e17-da4a-4f14-8397-f787cf5d1f11" (UID: "ae7e4e17-da4a-4f14-8397-f787cf5d1f11"). InnerVolumeSpecName "kube-api-access-nrt25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:16.800650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:16.800620 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrt25\" (UniqueName: \"kubernetes.io/projected/ae7e4e17-da4a-4f14-8397-f787cf5d1f11-kube-api-access-nrt25\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:17.367082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.367047 5073 generic.go:356] "Generic (PLEG): container finished" podID="ae7e4e17-da4a-4f14-8397-f787cf5d1f11" containerID="4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd" exitCode=0 Oct 31 00:10:17.367230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.367086 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" Oct 31 00:10:17.367230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.367136 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" event={"ID":"ae7e4e17-da4a-4f14-8397-f787cf5d1f11","Type":"ContainerDied","Data":"4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd"} Oct 31 00:10:17.367230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.367177 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr" event={"ID":"ae7e4e17-da4a-4f14-8397-f787cf5d1f11","Type":"ContainerDied","Data":"046700a068dfc85a26308e75e1c773f754c471c705d8e65cf56d0f4e2d3ded43"} Oct 31 00:10:17.367230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.367194 5073 scope.go:122] "RemoveContainer" containerID="4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd" Oct 31 00:10:17.376183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.376167 5073 scope.go:122] "RemoveContainer" containerID="4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd" Oct 31 00:10:17.376473 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:17.376456 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd\": container with ID starting with 4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd not found: ID does not exist" containerID="4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd" Oct 31 00:10:17.376508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.376482 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd"} err="failed to get container status \"4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd\": rpc error: code = NotFound desc = could not find container \"4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd\": container with ID starting with 4db89640b69819324b787cdbd91f344122650f145cbe577cac6184e2aa1963fd not found: ID does not exist" Oct 31 00:10:17.387517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.387494 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr"] Oct 31 00:10:17.391536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:17.391515 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-lf5fr"] Oct 31 00:10:18.058865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:18.058825 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7e4e17-da4a-4f14-8397-f787cf5d1f11" path="/var/lib/kubelet/pods/ae7e4e17-da4a-4f14-8397-f787cf5d1f11/volumes" Oct 31 00:10:19.927405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:19.927376 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:19.933431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:19.933407 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.028734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.028708 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.028854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.028817 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.031998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.031979 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.123369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.123344 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.140675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.140656 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.380243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.380221 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:20.667452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:20.667351 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:10:20.670278 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:20.670250 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd311ed02_63b1_4c82_9c7c_019c23f82869.slice/crio-3b4fb45c758a4a57510eb8b3bb29151b39eb056331f70c0235c4d706f051fe2f WatchSource:0}: Error finding container 3b4fb45c758a4a57510eb8b3bb29151b39eb056331f70c0235c4d706f051fe2f: Status 404 returned error can't find the container with id 3b4fb45c758a4a57510eb8b3bb29151b39eb056331f70c0235c4d706f051fe2f Oct 31 00:10:21.389611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:21.389538 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerStarted","Data":"159e2fc044776298155eadf272ddf847d645ac2961c877fa371735d66425cf15"} Oct 31 00:10:21.390082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:21.389639 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerStarted","Data":"3b4fb45c758a4a57510eb8b3bb29151b39eb056331f70c0235c4d706f051fe2f"} Oct 31 00:10:25.408349 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:25.408318 5073 generic.go:356] "Generic (PLEG): container finished" podID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerID="159e2fc044776298155eadf272ddf847d645ac2961c877fa371735d66425cf15" exitCode=0 Oct 31 00:10:25.408758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:25.408386 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerDied","Data":"159e2fc044776298155eadf272ddf847d645ac2961c877fa371735d66425cf15"} Oct 31 00:10:26.415053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:26.414970 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerStarted","Data":"e50a7133522a80ac87d42577498d3ca39398be0fe76ef90e6961ce3b4f22d0da"} Oct 31 00:10:26.415053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:26.415047 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerStarted","Data":"15e32316c32417c5f992408d0f10b3ffe36e24a8d0480762ddcdc865d2405028"} Oct 31 00:10:26.415436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:26.415073 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerStarted","Data":"f75f91e1075ce1e4bff9b9d3506158b6ab10e015cd1e2f94a72c802634b78bae"} Oct 31 00:10:26.447384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:26.447334 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.9639731430000005 podStartE2EDuration="10.447318046s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.483344903s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:26.445452587 +0000 UTC m=+710.506227315" watchObservedRunningTime="2026-10-31 00:10:26.447318046 +0000 UTC m=+710.508092741" Oct 31 00:10:30.381084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:30.381051 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:35.380707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:35.380669 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:35.385813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:35.385765 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:35.459304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:35.459279 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:43.085714 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:43.085677 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:10:48.281461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.281427 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:48.281886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.281619 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" containerName="otc-container" containerID="cri-o://4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926" gracePeriod=30 Oct 31 00:10:48.383261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.383230 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs"] Oct 31 00:10:48.383608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.383595 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ae7e4e17-da4a-4f14-8397-f787cf5d1f11" containerName="thanos-querier" Oct 31 00:10:48.383672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.383608 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7e4e17-da4a-4f14-8397-f787cf5d1f11" containerName="thanos-querier" Oct 31 00:10:48.383716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.383672 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7e4e17-da4a-4f14-8397-f787cf5d1f11" containerName="thanos-querier" Oct 31 00:10:48.386621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.386603 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.396315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.396282 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs"] Oct 31 00:10:48.455493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.455465 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzxk\" (UniqueName: \"kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.455661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.455507 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.455661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.455630 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.556109 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.556046 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.556109 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.556105 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.556295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.556144 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzxk\" (UniqueName: \"kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.556704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.556681 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.558757 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.558728 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.569084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.569056 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzxk\" (UniqueName: \"kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk\") pod \"data-science-collector-targetallocator-6c47f6cc5f-8p8vs\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.700699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.700680 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:48.977483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.977461 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs"] Oct 31 00:10:48.980061 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:48.980018 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a544d52_f11c_409a_880d_194cbd927987.slice/crio-637cc3e8617a43be66687d32b0782f9d1c83805a19552b6ccc2884b9abf407dd WatchSource:0}: Error finding container 637cc3e8617a43be66687d32b0782f9d1c83805a19552b6ccc2884b9abf407dd: Status 404 returned error can't find the container with id 637cc3e8617a43be66687d32b0782f9d1c83805a19552b6ccc2884b9abf407dd Oct 31 00:10:48.982094 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.982058 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:48.982189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:48.982117 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:49.515165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.515139 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.519967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.519941 5073 generic.go:356] "Generic (PLEG): container finished" podID="3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" containerID="4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926" exitCode=0 Oct 31 00:10:49.520070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.519985 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.520070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.520023 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7","Type":"ContainerDied","Data":"4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926"} Oct 31 00:10:49.520070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.520066 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7","Type":"ContainerDied","Data":"41964f83b2bc8d3f01aa482574a5edf5659235a73e27bbd1118b90d9725f739d"} Oct 31 00:10:49.520222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.520090 5073 scope.go:122] "RemoveContainer" containerID="4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926" Oct 31 00:10:49.521596 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.521577 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" event={"ID":"2a544d52-f11c-409a-880d-194cbd927987","Type":"ContainerStarted","Data":"58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c"} Oct 31 00:10:49.521684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.521602 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" event={"ID":"2a544d52-f11c-409a-880d-194cbd927987","Type":"ContainerStarted","Data":"637cc3e8617a43be66687d32b0782f9d1c83805a19552b6ccc2884b9abf407dd"} Oct 31 00:10:49.521737 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.521723 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:49.523450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.523430 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:10:49.528449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.528432 5073 scope.go:122] "RemoveContainer" containerID="4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926" Oct 31 00:10:49.528736 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:49.528715 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926\": container with ID starting with 4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926 not found: ID does not exist" containerID="4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926" Oct 31 00:10:49.528821 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.528749 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926"} err="failed to get container status \"4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926\": rpc error: code = NotFound desc = could not find container \"4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926\": container with ID starting with 4d334b3b6e9cd56f3df2ab50d5d43c40338ee9c7877cfa46dbc93b3dee466926 not found: ID does not exist" Oct 31 00:10:49.562562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.562528 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs\") pod \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " Oct 31 00:10:49.562562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.562560 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert\") pod \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " Oct 31 00:10:49.562763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.562600 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\") pod \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " Oct 31 00:10:49.562763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.562658 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal\") pod \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " Oct 31 00:10:49.562763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.562690 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs\") pod \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " Oct 31 00:10:49.563627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.562807 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs\" (UniqueName: \"kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs\") pod \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\" (UID: \"3ee9b82c-2498-4b77-8a6d-f08c6c0299e7\") " Oct 31 00:10:49.564602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.564569 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal" pod "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:49.565280 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.565253 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:49.565698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.565649 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" podStartSLOduration=1.565634465 podStartE2EDuration="1.565634465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:49.563420474 +0000 UTC m=+733.624195170" watchObservedRunningTime="2026-10-31 00:10:49.565634465 +0000 UTC m=+733.626409151" Oct 31 00:10:49.566279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.566237 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert" pod "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:49.566279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.566265 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs" pod "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:49.567285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.566998 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs" pod "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7"). InnerVolumeSpecName "kube-api-access-pxqbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:49.567553 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.567430 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs" pod "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:49.568354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.568318 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs" pod "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" (UID: "3ee9b82c-2498-4b77-8a6d-f08c6c0299e7"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:49.593186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.593163 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc"] Oct 31 00:10:49.593367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.593332 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" podUID="e00a0ce9-8ed0-403a-9946-7aa0973b480b" containerName="ta-container" containerID="cri-o://a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f" gracePeriod=30 Oct 31 00:10:49.666046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.666014 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqbs\" (UniqueName: \"kubernetes.io/projected/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-kube-api-access-pxqbs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:49.666046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.666045 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:49.666190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.666061 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:49.666190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.666074 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:49.666190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.666087 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:49.842915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.842893 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:49.853276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.853254 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:49.859080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.859060 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:49.888744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.888679 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:49.889115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.889098 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" containerName="otc-container" Oct 31 00:10:49.889189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.889118 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" containerName="otc-container" Oct 31 00:10:49.889189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.889139 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e00a0ce9-8ed0-403a-9946-7aa0973b480b" containerName="ta-container" Oct 31 00:10:49.889189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.889147 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00a0ce9-8ed0-403a-9946-7aa0973b480b" containerName="ta-container" Oct 31 00:10:49.889353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.889235 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" containerName="otc-container" Oct 31 00:10:49.889353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.889250 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00a0ce9-8ed0-403a-9946-7aa0973b480b" containerName="ta-container" Oct 31 00:10:49.892204 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.892185 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.896216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.896198 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:10:49.896769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.896751 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:10:49.896897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.896810 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:10:49.896897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.896852 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:10:49.896897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.896854 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:10:49.896897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.896883 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:10:49.904436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.904407 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:49.968438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968413 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert\") pod \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " Oct 31 00:10:49.968573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968496 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq\" (UniqueName: \"kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq\") pod \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " Oct 31 00:10:49.968573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968523 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal\" (UniqueName: \"kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal\") pod \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\" (UID: \"e00a0ce9-8ed0-403a-9946-7aa0973b480b\") " Oct 31 00:10:49.968694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968605 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.968694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968629 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnp82\" (UniqueName: \"kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.968694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968655 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.968881 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968824 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.968948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968934 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.968988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968966 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:49.969026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.968985 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal" pod "e00a0ce9-8ed0-403a-9946-7aa0973b480b" (UID: "e00a0ce9-8ed0-403a-9946-7aa0973b480b"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:49.970682 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.970660 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert" pod "e00a0ce9-8ed0-403a-9946-7aa0973b480b" (UID: "e00a0ce9-8ed0-403a-9946-7aa0973b480b"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:49.970749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:49.970697 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq" pod "e00a0ce9-8ed0-403a-9946-7aa0973b480b" (UID: "e00a0ce9-8ed0-403a-9946-7aa0973b480b"). InnerVolumeSpecName "kube-api-access-4lxpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:50.059202 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.059177 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee9b82c-2498-4b77-8a6d-f08c6c0299e7" path="/var/lib/kubelet/pods/3ee9b82c-2498-4b77-8a6d-f08c6c0299e7/volumes" Oct 31 00:10:50.070153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070134 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.070251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070168 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.070251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070220 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.070251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070237 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.070355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070268 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.070412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070388 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnp82\" (UniqueName: \"kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.070494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070477 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lxpq\" (UniqueName: \"kubernetes.io/projected/e00a0ce9-8ed0-403a-9946-7aa0973b480b-kube-api-access-4lxpq\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:50.070546 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070501 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/e00a0ce9-8ed0-403a-9946-7aa0973b480b-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:50.070546 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.070518 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/e00a0ce9-8ed0-403a-9946-7aa0973b480b-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:50.071242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.071220 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.072629 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.072607 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.072706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.072643 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.073082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.073065 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.073117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.073075 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.082390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.082370 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnp82\" (UniqueName: \"kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82\") pod \"data-science-collector-collector-1\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.208411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.208350 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:10:50.485522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.485492 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:10:50.489199 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:10:50.489171 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb85068_59dc_481d_9b98_ad88c4267458.slice/crio-6fd8531aca5277e7cffaa145d1da9b21347cdd4d19e2c3bc159ade70889c40d5 WatchSource:0}: Error finding container 6fd8531aca5277e7cffaa145d1da9b21347cdd4d19e2c3bc159ade70889c40d5: Status 404 returned error can't find the container with id 6fd8531aca5277e7cffaa145d1da9b21347cdd4d19e2c3bc159ade70889c40d5 Oct 31 00:10:50.491451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.491417 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:50.491535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.491480 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:10:50.529124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.529094 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bfb85068-59dc-481d-9b98-ad88c4267458","Type":"ContainerStarted","Data":"6fd8531aca5277e7cffaa145d1da9b21347cdd4d19e2c3bc159ade70889c40d5"} Oct 31 00:10:50.531683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.531656 5073 generic.go:356] "Generic (PLEG): container finished" podID="e00a0ce9-8ed0-403a-9946-7aa0973b480b" containerID="a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f" exitCode=0 Oct 31 00:10:50.532388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.532369 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" Oct 31 00:10:50.532759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.532736 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" event={"ID":"e00a0ce9-8ed0-403a-9946-7aa0973b480b","Type":"ContainerDied","Data":"a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f"} Oct 31 00:10:50.532898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.532770 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc" event={"ID":"e00a0ce9-8ed0-403a-9946-7aa0973b480b","Type":"ContainerDied","Data":"b6157b9fecc96e0684e65baa08ac83ba368ef91dafb909edf64465dcad8d1b22"} Oct 31 00:10:50.532898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.532813 5073 scope.go:122] "RemoveContainer" containerID="a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f" Oct 31 00:10:50.543067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.543043 5073 scope.go:122] "RemoveContainer" containerID="a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f" Oct 31 00:10:50.543322 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:10:50.543305 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f\": container with ID starting with a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f not found: ID does not exist" containerID="a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f" Oct 31 00:10:50.543385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.543333 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f"} err="failed to get container status \"a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f\": rpc error: code = NotFound desc = could not find container \"a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f\": container with ID starting with a7c735d3eb736f3fbbb333df071fc6c926e066c94f5a8d12917a8fe2e79a8f2f not found: ID does not exist" Oct 31 00:10:50.551974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.551951 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc"] Oct 31 00:10:50.556473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:50.556452 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-859d7c9d9d-svsfc"] Oct 31 00:10:51.538601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:51.538569 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bfb85068-59dc-481d-9b98-ad88c4267458","Type":"ContainerStarted","Data":"03314a2b91428b8474b565db44fee68216e0a89599c22ba870ea0a2f83a60e8e"} Oct 31 00:10:51.559994 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:51.559936 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.55991797 podStartE2EDuration="2.55991797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:51.558123231 +0000 UTC m=+735.618897926" watchObservedRunningTime="2026-10-31 00:10:51.55991797 +0000 UTC m=+735.620692666" Oct 31 00:10:52.058366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:10:52.058326 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00a0ce9-8ed0-403a-9946-7aa0973b480b" path="/var/lib/kubelet/pods/e00a0ce9-8ed0-403a-9946-7aa0973b480b/volumes" Oct 31 00:11:13.087256 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:11:13.087223 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:11:43.085917 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:11:43.085884 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:12:01.453246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.453213 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619"] Oct 31 00:12:01.456772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.456755 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" Oct 31 00:12:01.464643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.463990 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619"] Oct 31 00:12:01.468948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.468930 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" Oct 31 00:12:01.745185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.745071 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619"] Oct 31 00:12:01.748139 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:01.748115 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21efa13a_549d_4da5_a90c_e1e75bd84513.slice/crio-fb65d2332a5ba0d90dc26b7609ef631556a0fdb7d0249fa674a28170870b898d WatchSource:0}: Error finding container fb65d2332a5ba0d90dc26b7609ef631556a0fdb7d0249fa674a28170870b898d: Status 404 returned error can't find the container with id fb65d2332a5ba0d90dc26b7609ef631556a0fdb7d0249fa674a28170870b898d Oct 31 00:12:01.750690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.750672 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:12:01.865757 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:01.865732 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" event={"ID":"21efa13a-549d-4da5-a90c-e1e75bd84513","Type":"ContainerStarted","Data":"fb65d2332a5ba0d90dc26b7609ef631556a0fdb7d0249fa674a28170870b898d"} Oct 31 00:12:02.871388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:02.871311 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" event={"ID":"21efa13a-549d-4da5-a90c-e1e75bd84513","Type":"ContainerStarted","Data":"856e5fd43c2edfa5f6cfb6fbcae6a907336384c29a0ca3ccf4f641a6664f33bc"} Oct 31 00:12:02.887434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:02.887373 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" podStartSLOduration=1.016702146 podStartE2EDuration="1.887356627s" totalImagesPullingTime="870.654481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:02.886982438 +0000 UTC m=+806.947757134" watchObservedRunningTime="2026-10-31 00:12:02.887356627 +0000 UTC m=+806.948131322" Oct 31 00:12:07.896483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:07.896456 5073 generic.go:356] "Generic (PLEG): container finished" podID="21efa13a-549d-4da5-a90c-e1e75bd84513" containerID="856e5fd43c2edfa5f6cfb6fbcae6a907336384c29a0ca3ccf4f641a6664f33bc" exitCode=0 Oct 31 00:12:07.896829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:07.896541 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" event={"ID":"21efa13a-549d-4da5-a90c-e1e75bd84513","Type":"ContainerDied","Data":"856e5fd43c2edfa5f6cfb6fbcae6a907336384c29a0ca3ccf4f641a6664f33bc"} Oct 31 00:12:09.029926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:09.029901 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" Oct 31 00:12:09.530538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:09.530511 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619"] Oct 31 00:12:09.532924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:09.532901 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619"] Oct 31 00:12:09.906686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:09.906606 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" Oct 31 00:12:09.906686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:09.906650 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb65d2332a5ba0d90dc26b7609ef631556a0fdb7d0249fa674a28170870b898d" Oct 31 00:12:09.916404 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:09.916370 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"korrel8r-ingress-18e374d1e23ee619\" is forbidden: User \"system:node:ip-10-0-140-246.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-140-246.us-east-2.compute.internal' and this object" podUID="21efa13a-549d-4da5-a90c-e1e75bd84513" pod="redhat-ods-monitoring/korrel8r-ingress-18e374d1e23ee619" Oct 31 00:12:10.058915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:10.058886 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21efa13a-549d-4da5-a90c-e1e75bd84513" path="/var/lib/kubelet/pods/21efa13a-549d-4da5-a90c-e1e75bd84513/volumes" Oct 31 00:12:13.084657 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:13.084629 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:12:16.725035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.724998 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:16.725440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.725190 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="bfb85068-59dc-481d-9b98-ad88c4267458" containerName="otc-container" containerID="cri-o://03314a2b91428b8474b565db44fee68216e0a89599c22ba870ea0a2f83a60e8e" gracePeriod=30 Oct 31 00:12:16.830729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.830693 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx"] Oct 31 00:12:16.831114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.831099 5073 cpu_manager.go:414] "containerMap: removing container" podUID="21efa13a-549d-4da5-a90c-e1e75bd84513" containerName="curl" Oct 31 00:12:16.831170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.831115 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21efa13a-549d-4da5-a90c-e1e75bd84513" containerName="curl" Oct 31 00:12:16.831214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.831205 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="21efa13a-549d-4da5-a90c-e1e75bd84513" containerName="curl" Oct 31 00:12:16.834203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.834188 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:16.844295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.844256 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx"] Oct 31 00:12:16.955620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.955586 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:16.955816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.955645 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:16.955816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:16.955733 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crsh\" (UniqueName: \"kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.056337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.056308 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.056493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.056353 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.056493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.056383 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crsh\" (UniqueName: \"kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.056941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.056919 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.058916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.058878 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.068804 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.068763 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crsh\" (UniqueName: \"kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh\") pod \"data-science-collector-targetallocator-64d554cdcb-b6qmx\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.147933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.147906 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.436311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.436234 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx"] Oct 31 00:12:17.439421 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:17.439381 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73d81019_1458_4b13_89bb_6967c062dac8.slice/crio-06da3916d9ff2616bff5ee9b15fc4c7eda954ceda6331cb0d5e518a73cf72fd3 WatchSource:0}: Error finding container 06da3916d9ff2616bff5ee9b15fc4c7eda954ceda6331cb0d5e518a73cf72fd3: Status 404 returned error can't find the container with id 06da3916d9ff2616bff5ee9b15fc4c7eda954ceda6331cb0d5e518a73cf72fd3 Oct 31 00:12:17.441620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.441586 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:17.441705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.441641 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:17.943551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.943511 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" event={"ID":"73d81019-1458-4b13-89bb-6967c062dac8","Type":"ContainerStarted","Data":"7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa"} Oct 31 00:12:17.943965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.943565 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" event={"ID":"73d81019-1458-4b13-89bb-6967c062dac8","Type":"ContainerStarted","Data":"06da3916d9ff2616bff5ee9b15fc4c7eda954ceda6331cb0d5e518a73cf72fd3"} Oct 31 00:12:17.943965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.943586 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.945434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.945410 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:17.963811 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:17.963753 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" podStartSLOduration=1.9637407420000002 podStartE2EDuration="1.963740742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:17.961726458 +0000 UTC m=+822.022501154" watchObservedRunningTime="2026-10-31 00:12:17.963740742 +0000 UTC m=+822.024515437" Oct 31 00:12:18.012635 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.012606 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs"] Oct 31 00:12:18.012817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.012795 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" podUID="2a544d52-f11c-409a-880d-194cbd927987" containerName="ta-container" containerID="cri-o://58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c" gracePeriod=30 Oct 31 00:12:18.257115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.257091 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:12:18.369164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.369136 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal\" (UniqueName: \"kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal\") pod \"2a544d52-f11c-409a-880d-194cbd927987\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " Oct 31 00:12:18.369309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.369193 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk\" (UniqueName: \"kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk\") pod \"2a544d52-f11c-409a-880d-194cbd927987\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " Oct 31 00:12:18.369309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.369223 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert\") pod \"2a544d52-f11c-409a-880d-194cbd927987\" (UID: \"2a544d52-f11c-409a-880d-194cbd927987\") " Oct 31 00:12:18.369472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.369452 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal" pod "2a544d52-f11c-409a-880d-194cbd927987" (UID: "2a544d52-f11c-409a-880d-194cbd927987"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:18.371511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.371485 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk" pod "2a544d52-f11c-409a-880d-194cbd927987" (UID: "2a544d52-f11c-409a-880d-194cbd927987"). InnerVolumeSpecName "kube-api-access-bmzxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:18.371511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.371492 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert" pod "2a544d52-f11c-409a-880d-194cbd927987" (UID: "2a544d52-f11c-409a-880d-194cbd927987"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:18.470569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.470544 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2a544d52-f11c-409a-880d-194cbd927987-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:18.470569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.470567 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzxk\" (UniqueName: \"kubernetes.io/projected/2a544d52-f11c-409a-880d-194cbd927987-kube-api-access-bmzxk\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:18.470738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.470579 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2a544d52-f11c-409a-880d-194cbd927987-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:18.955726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.955688 5073 generic.go:356] "Generic (PLEG): container finished" podID="2a544d52-f11c-409a-880d-194cbd927987" containerID="58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c" exitCode=0 Oct 31 00:12:18.956179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.955728 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" event={"ID":"2a544d52-f11c-409a-880d-194cbd927987","Type":"ContainerDied","Data":"58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c"} Oct 31 00:12:18.956179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.955889 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" event={"ID":"2a544d52-f11c-409a-880d-194cbd927987","Type":"ContainerDied","Data":"637cc3e8617a43be66687d32b0782f9d1c83805a19552b6ccc2884b9abf407dd"} Oct 31 00:12:18.956179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.955919 5073 scope.go:122] "RemoveContainer" containerID="58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c" Oct 31 00:12:18.956179 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:18.955923 5073 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb85068_59dc_481d_9b98_ad88c4267458.slice/crio-03314a2b91428b8474b565db44fee68216e0a89599c22ba870ea0a2f83a60e8e.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:12:18.956179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.955938 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs" Oct 31 00:12:18.958522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.958498 5073 generic.go:356] "Generic (PLEG): container finished" podID="bfb85068-59dc-481d-9b98-ad88c4267458" containerID="03314a2b91428b8474b565db44fee68216e0a89599c22ba870ea0a2f83a60e8e" exitCode=0 Oct 31 00:12:18.958665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.958636 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bfb85068-59dc-481d-9b98-ad88c4267458","Type":"ContainerDied","Data":"03314a2b91428b8474b565db44fee68216e0a89599c22ba870ea0a2f83a60e8e"} Oct 31 00:12:18.967319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.967297 5073 scope.go:122] "RemoveContainer" containerID="58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c" Oct 31 00:12:18.967585 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:18.967566 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c\": container with ID starting with 58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c not found: ID does not exist" containerID="58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c" Oct 31 00:12:18.967664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.967592 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c"} err="failed to get container status \"58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c\": rpc error: code = NotFound desc = could not find container \"58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c\": container with ID starting with 58def0eefa9673d0fb758078a070f2c8fee8bae94ffb1db52a7e0c306400057c not found: ID does not exist" Oct 31 00:12:18.979934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.979908 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs"] Oct 31 00:12:18.984870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:18.984847 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c47f6cc5f-8p8vs"] Oct 31 00:12:19.064179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.064158 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:19.175581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.175519 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs\") pod \"bfb85068-59dc-481d-9b98-ad88c4267458\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " Oct 31 00:12:19.175700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.175578 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82\" (UniqueName: \"kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82\") pod \"bfb85068-59dc-481d-9b98-ad88c4267458\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " Oct 31 00:12:19.175700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.175684 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs\") pod \"bfb85068-59dc-481d-9b98-ad88c4267458\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " Oct 31 00:12:19.175816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.175713 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs\") pod \"bfb85068-59dc-481d-9b98-ad88c4267458\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " Oct 31 00:12:19.175816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.175746 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal\" (UniqueName: \"kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal\") pod \"bfb85068-59dc-481d-9b98-ad88c4267458\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " Oct 31 00:12:19.175936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.175813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert\") pod \"bfb85068-59dc-481d-9b98-ad88c4267458\" (UID: \"bfb85068-59dc-481d-9b98-ad88c4267458\") " Oct 31 00:12:19.176206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.176179 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal" pod "bfb85068-59dc-481d-9b98-ad88c4267458" (UID: "bfb85068-59dc-481d-9b98-ad88c4267458"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:19.177834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.177748 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs" pod "bfb85068-59dc-481d-9b98-ad88c4267458" (UID: "bfb85068-59dc-481d-9b98-ad88c4267458"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:19.177978 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.177887 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs" pod "bfb85068-59dc-481d-9b98-ad88c4267458" (UID: "bfb85068-59dc-481d-9b98-ad88c4267458"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:19.178213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.178189 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert" pod "bfb85068-59dc-481d-9b98-ad88c4267458" (UID: "bfb85068-59dc-481d-9b98-ad88c4267458"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:19.178213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.178205 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs" pod "bfb85068-59dc-481d-9b98-ad88c4267458" (UID: "bfb85068-59dc-481d-9b98-ad88c4267458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:19.178332 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.178207 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82" pod "bfb85068-59dc-481d-9b98-ad88c4267458" (UID: "bfb85068-59dc-481d-9b98-ad88c4267458"). InnerVolumeSpecName "kube-api-access-gnp82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:19.277370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.277341 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:19.277370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.277367 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnp82\" (UniqueName: \"kubernetes.io/projected/bfb85068-59dc-481d-9b98-ad88c4267458-kube-api-access-gnp82\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:19.277554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.277380 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:19.277554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.277389 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:19.277554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.277398 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bfb85068-59dc-481d-9b98-ad88c4267458-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:19.277554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.277407 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/bfb85068-59dc-481d-9b98-ad88c4267458-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:19.964499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.964471 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:19.965022 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.964522 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"bfb85068-59dc-481d-9b98-ad88c4267458","Type":"ContainerDied","Data":"6fd8531aca5277e7cffaa145d1da9b21347cdd4d19e2c3bc159ade70889c40d5"} Oct 31 00:12:19.965022 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.964596 5073 scope.go:122] "RemoveContainer" containerID="03314a2b91428b8474b565db44fee68216e0a89599c22ba870ea0a2f83a60e8e" Oct 31 00:12:19.986007 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.985983 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:19.990806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:19.990768 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:20.016588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.016562 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:20.016949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.016936 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2a544d52-f11c-409a-880d-194cbd927987" containerName="ta-container" Oct 31 00:12:20.017011 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.016949 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a544d52-f11c-409a-880d-194cbd927987" containerName="ta-container" Oct 31 00:12:20.017011 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.016966 5073 cpu_manager.go:414] "containerMap: removing container" podUID="bfb85068-59dc-481d-9b98-ad88c4267458" containerName="otc-container" Oct 31 00:12:20.017011 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.016971 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb85068-59dc-481d-9b98-ad88c4267458" containerName="otc-container" Oct 31 00:12:20.017111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.017022 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a544d52-f11c-409a-880d-194cbd927987" containerName="ta-container" Oct 31 00:12:20.017111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.017031 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb85068-59dc-481d-9b98-ad88c4267458" containerName="otc-container" Oct 31 00:12:20.021341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.021327 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.025579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.025560 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:20.025579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.025560 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:20.025710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.025585 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:20.025710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.025663 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:20.025710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.025706 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:20.025849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.025757 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:12:20.030931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.030912 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:20.057862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.057833 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a544d52-f11c-409a-880d-194cbd927987" path="/var/lib/kubelet/pods/2a544d52-f11c-409a-880d-194cbd927987/volumes" Oct 31 00:12:20.058201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.058186 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb85068-59dc-481d-9b98-ad88c4267458" path="/var/lib/kubelet/pods/bfb85068-59dc-481d-9b98-ad88c4267458/volumes" Oct 31 00:12:20.187003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.186969 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.187119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.187013 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.187170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.187127 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.187216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.187200 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.187270 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.187259 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.187322 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.187282 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxwx\" (UniqueName: \"kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.288295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.288271 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.288421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.288310 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxwx\" (UniqueName: \"kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.288421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.288345 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.288421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.288379 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.288622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.288427 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.288622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.288475 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.289077 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.289047 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.291212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.291185 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.291306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.291185 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.291306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.291266 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.291376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.291355 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.301247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.301225 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxwx\" (UniqueName: \"kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx\") pod \"data-science-collector-collector-1\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.335021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.334993 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:20.610939 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.610725 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:20.613657 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:20.613629 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a529b7_6aea_46da_857b_4ec91d53d098.slice/crio-270798ef6f1dbf387db27a7aaa8246a63fc551be492040251aa5e204db854710 WatchSource:0}: Error finding container 270798ef6f1dbf387db27a7aaa8246a63fc551be492040251aa5e204db854710: Status 404 returned error can't find the container with id 270798ef6f1dbf387db27a7aaa8246a63fc551be492040251aa5e204db854710 Oct 31 00:12:20.615834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.615798 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:20.615906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.615862 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:20.973150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.973063 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"58a529b7-6aea-46da-857b-4ec91d53d098","Type":"ContainerStarted","Data":"03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81"} Oct 31 00:12:20.973150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.973110 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"58a529b7-6aea-46da-857b-4ec91d53d098","Type":"ContainerStarted","Data":"270798ef6f1dbf387db27a7aaa8246a63fc551be492040251aa5e204db854710"} Oct 31 00:12:20.995235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:20.995186 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.995174086 podStartE2EDuration="1.995174086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:20.993411315 +0000 UTC m=+825.054186023" watchObservedRunningTime="2026-10-31 00:12:20.995174086 +0000 UTC m=+825.055948781" Oct 31 00:12:29.252923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.252890 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:29.253299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.253101 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="58a529b7-6aea-46da-857b-4ec91d53d098" containerName="otc-container" containerID="cri-o://03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81" gracePeriod=30 Oct 31 00:12:29.352292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.352262 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7"] Oct 31 00:12:29.355841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.355822 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.367166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.367139 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7"] Oct 31 00:12:29.464342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.464315 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.464483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.464379 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5mzs\" (UniqueName: \"kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.464521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.464484 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.512314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.512263 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:29.565205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.565176 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5mzs\" (UniqueName: \"kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.565365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.565230 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.565365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.565287 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.566087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.566055 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.568073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.568039 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.580674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.580653 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5mzs\" (UniqueName: \"kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs\") pod \"data-science-collector-targetallocator-c75f89845-6nzf7\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.666236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666212 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert\") pod \"58a529b7-6aea-46da-857b-4ec91d53d098\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " Oct 31 00:12:29.666360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666283 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs\") pod \"58a529b7-6aea-46da-857b-4ec91d53d098\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " Oct 31 00:12:29.666360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666323 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs\") pod \"58a529b7-6aea-46da-857b-4ec91d53d098\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " Oct 31 00:12:29.666360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666352 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal\" (UniqueName: \"kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal\") pod \"58a529b7-6aea-46da-857b-4ec91d53d098\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " Oct 31 00:12:29.666549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666372 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs\") pod \"58a529b7-6aea-46da-857b-4ec91d53d098\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " Oct 31 00:12:29.666549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666433 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx\" (UniqueName: \"kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx\") pod \"58a529b7-6aea-46da-857b-4ec91d53d098\" (UID: \"58a529b7-6aea-46da-857b-4ec91d53d098\") " Oct 31 00:12:29.666709 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.666676 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal" pod "58a529b7-6aea-46da-857b-4ec91d53d098" (UID: "58a529b7-6aea-46da-857b-4ec91d53d098"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:29.668443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.668411 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert" pod "58a529b7-6aea-46da-857b-4ec91d53d098" (UID: "58a529b7-6aea-46da-857b-4ec91d53d098"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:29.668564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.668540 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs" pod "58a529b7-6aea-46da-857b-4ec91d53d098" (UID: "58a529b7-6aea-46da-857b-4ec91d53d098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:29.668894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.668870 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx" pod "58a529b7-6aea-46da-857b-4ec91d53d098" (UID: "58a529b7-6aea-46da-857b-4ec91d53d098"). InnerVolumeSpecName "kube-api-access-chxwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:29.669006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.668984 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs" pod "58a529b7-6aea-46da-857b-4ec91d53d098" (UID: "58a529b7-6aea-46da-857b-4ec91d53d098"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:29.669049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.668996 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs" pod "58a529b7-6aea-46da-857b-4ec91d53d098" (UID: "58a529b7-6aea-46da-857b-4ec91d53d098"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:29.671164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.671146 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:29.767416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.767337 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:29.767416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.767367 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:29.767416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.767376 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/58a529b7-6aea-46da-857b-4ec91d53d098-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:29.767416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.767385 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:29.767416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.767394 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxwx\" (UniqueName: \"kubernetes.io/projected/58a529b7-6aea-46da-857b-4ec91d53d098-kube-api-access-chxwx\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:29.767416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.767405 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/58a529b7-6aea-46da-857b-4ec91d53d098-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:29.947059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.946988 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7"] Oct 31 00:12:29.949773 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:29.949750 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a3d11c_1326_4033_b3d6_c7f8d0279069.slice/crio-c2b65d33a3da00632c8ad5657f996cec0097d1037b294cc3ae9393c976774f40 WatchSource:0}: Error finding container c2b65d33a3da00632c8ad5657f996cec0097d1037b294cc3ae9393c976774f40: Status 404 returned error can't find the container with id c2b65d33a3da00632c8ad5657f996cec0097d1037b294cc3ae9393c976774f40 Oct 31 00:12:29.951811 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.951757 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:29.951861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:29.951831 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:30.023428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.023362 5073 generic.go:356] "Generic (PLEG): container finished" podID="58a529b7-6aea-46da-857b-4ec91d53d098" containerID="03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81" exitCode=0 Oct 31 00:12:30.023428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.023409 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.023600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.023437 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"58a529b7-6aea-46da-857b-4ec91d53d098","Type":"ContainerDied","Data":"03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81"} Oct 31 00:12:30.023600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.023492 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"58a529b7-6aea-46da-857b-4ec91d53d098","Type":"ContainerDied","Data":"270798ef6f1dbf387db27a7aaa8246a63fc551be492040251aa5e204db854710"} Oct 31 00:12:30.023600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.023513 5073 scope.go:122] "RemoveContainer" containerID="03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81" Oct 31 00:12:30.025417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.025388 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" event={"ID":"c8a3d11c-1326-4033-b3d6-c7f8d0279069","Type":"ContainerStarted","Data":"974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9"} Oct 31 00:12:30.025535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.025430 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" event={"ID":"c8a3d11c-1326-4033-b3d6-c7f8d0279069","Type":"ContainerStarted","Data":"c2b65d33a3da00632c8ad5657f996cec0097d1037b294cc3ae9393c976774f40"} Oct 31 00:12:30.025664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.025633 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:30.027716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.027685 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" podUID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.65:8080/readyz\": dial tcp 10.132.0.65:8080: connect: connection refused" Oct 31 00:12:30.033763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.033741 5073 scope.go:122] "RemoveContainer" containerID="03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81" Oct 31 00:12:30.034059 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:30.034042 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81\": container with ID starting with 03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81 not found: ID does not exist" containerID="03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81" Oct 31 00:12:30.034136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.034073 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81"} err="failed to get container status \"03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81\": rpc error: code = NotFound desc = could not find container \"03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81\": container with ID starting with 03e7ab3ced39973230937f597226be94a431f4acdccc4085161eebf61d033d81 not found: ID does not exist" Oct 31 00:12:30.048439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.048387 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" podStartSLOduration=1.048376195 podStartE2EDuration="1.048376195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:30.047852011 +0000 UTC m=+834.108626709" watchObservedRunningTime="2026-10-31 00:12:30.048376195 +0000 UTC m=+834.109150889" Oct 31 00:12:30.075110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.075087 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:30.079597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.079575 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:30.113348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.113325 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:30.113667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.113654 5073 cpu_manager.go:414] "containerMap: removing container" podUID="58a529b7-6aea-46da-857b-4ec91d53d098" containerName="otc-container" Oct 31 00:12:30.113729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.113667 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a529b7-6aea-46da-857b-4ec91d53d098" containerName="otc-container" Oct 31 00:12:30.113817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.113740 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a529b7-6aea-46da-857b-4ec91d53d098" containerName="otc-container" Oct 31 00:12:30.118118 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.118101 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.127819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.127799 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:30.128101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.127984 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:12:30.128101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.127997 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:30.128101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.128047 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:30.128101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.128066 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:30.128805 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.128123 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:30.129409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.129356 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:30.171373 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.171349 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.171473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.171376 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jfp\" (UniqueName: \"kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.171473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.171401 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.171551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.171514 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.171551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.171543 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.171618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.171579 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.272818 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.272795 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.273195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.272826 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jfp\" (UniqueName: \"kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.273195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.272851 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.273195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.272910 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.273195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.273157 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.273398 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.273220 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.274203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.274145 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.276283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.276241 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.276283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.276251 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.276381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.276341 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.276423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.276402 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.285473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.285454 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jfp\" (UniqueName: \"kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp\") pod \"data-science-collector-collector-1\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.434347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.434323 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:30.711955 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.711932 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:30.714159 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:30.714133 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47046010_bdeb_41ca_aa81_2a6b6445641e.slice/crio-592660b41df978ac687b0566d86f688cce7236244b96468bd18ac8fb03e03631 WatchSource:0}: Error finding container 592660b41df978ac687b0566d86f688cce7236244b96468bd18ac8fb03e03631: Status 404 returned error can't find the container with id 592660b41df978ac687b0566d86f688cce7236244b96468bd18ac8fb03e03631 Oct 31 00:12:30.716346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.716312 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:30.716450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:30.716368 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:31.032095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.032059 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"47046010-bdeb-41ca-aa81-2a6b6445641e","Type":"ContainerStarted","Data":"d336e1fb20f6698b65379300caf9f4fd69a9d4c0b8e7d41cfc4f480f0ed7aab5"} Oct 31 00:12:31.032276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.032114 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"47046010-bdeb-41ca-aa81-2a6b6445641e","Type":"ContainerStarted","Data":"592660b41df978ac687b0566d86f688cce7236244b96468bd18ac8fb03e03631"} Oct 31 00:12:31.034159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.034135 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:31.053088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.053051 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.05303692 podStartE2EDuration="1.05303692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:31.051870889 +0000 UTC m=+835.112645585" watchObservedRunningTime="2026-10-31 00:12:31.05303692 +0000 UTC m=+835.113811615" Oct 31 00:12:31.101058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.101027 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx"] Oct 31 00:12:31.101215 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.101196 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" podUID="73d81019-1458-4b13-89bb-6967c062dac8" containerName="ta-container" containerID="cri-o://7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa" gracePeriod=30 Oct 31 00:12:31.349337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.349315 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:31.382441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.382415 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh\" (UniqueName: \"kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh\") pod \"73d81019-1458-4b13-89bb-6967c062dac8\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " Oct 31 00:12:31.382594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.382456 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal\" (UniqueName: \"kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal\") pod \"73d81019-1458-4b13-89bb-6967c062dac8\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " Oct 31 00:12:31.382594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.382483 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert\") pod \"73d81019-1458-4b13-89bb-6967c062dac8\" (UID: \"73d81019-1458-4b13-89bb-6967c062dac8\") " Oct 31 00:12:31.382909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.382872 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal" pod "73d81019-1458-4b13-89bb-6967c062dac8" (UID: "73d81019-1458-4b13-89bb-6967c062dac8"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:31.384651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.384621 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh" pod "73d81019-1458-4b13-89bb-6967c062dac8" (UID: "73d81019-1458-4b13-89bb-6967c062dac8"). InnerVolumeSpecName "kube-api-access-7crsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:31.384739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.384680 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert" pod "73d81019-1458-4b13-89bb-6967c062dac8" (UID: "73d81019-1458-4b13-89bb-6967c062dac8"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:31.483276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.483252 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7crsh\" (UniqueName: \"kubernetes.io/projected/73d81019-1458-4b13-89bb-6967c062dac8-kube-api-access-7crsh\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.483276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.483276 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/73d81019-1458-4b13-89bb-6967c062dac8-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.483424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:31.483286 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/73d81019-1458-4b13-89bb-6967c062dac8-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:32.039374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.039342 5073 generic.go:356] "Generic (PLEG): container finished" podID="73d81019-1458-4b13-89bb-6967c062dac8" containerID="7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa" exitCode=0 Oct 31 00:12:32.039532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.039397 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" Oct 31 00:12:32.039532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.039427 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" event={"ID":"73d81019-1458-4b13-89bb-6967c062dac8","Type":"ContainerDied","Data":"7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa"} Oct 31 00:12:32.039532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.039479 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx" event={"ID":"73d81019-1458-4b13-89bb-6967c062dac8","Type":"ContainerDied","Data":"06da3916d9ff2616bff5ee9b15fc4c7eda954ceda6331cb0d5e518a73cf72fd3"} Oct 31 00:12:32.039532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.039510 5073 scope.go:122] "RemoveContainer" containerID="7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa" Oct 31 00:12:32.049253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.049236 5073 scope.go:122] "RemoveContainer" containerID="7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa" Oct 31 00:12:32.049523 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:32.049504 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa\": container with ID starting with 7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa not found: ID does not exist" containerID="7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa" Oct 31 00:12:32.049565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.049531 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa"} err="failed to get container status \"7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa\": rpc error: code = NotFound desc = could not find container \"7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa\": container with ID starting with 7f0498ea7228dad5ad7888b9fa63490d4920a99b3f8fabb0377cc92fe62581aa not found: ID does not exist" Oct 31 00:12:32.058368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.058345 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a529b7-6aea-46da-857b-4ec91d53d098" path="/var/lib/kubelet/pods/58a529b7-6aea-46da-857b-4ec91d53d098/volumes" Oct 31 00:12:32.060297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.060275 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx"] Oct 31 00:12:32.064304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:32.064285 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-64d554cdcb-b6qmx"] Oct 31 00:12:34.058477 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:34.058440 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d81019-1458-4b13-89bb-6967c062dac8" path="/var/lib/kubelet/pods/73d81019-1458-4b13-89bb-6967c062dac8/volumes" Oct 31 00:12:41.815464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.815424 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:41.815966 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.815739 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="47046010-bdeb-41ca-aa81-2a6b6445641e" containerName="otc-container" containerID="cri-o://d336e1fb20f6698b65379300caf9f4fd69a9d4c0b8e7d41cfc4f480f0ed7aab5" gracePeriod=30 Oct 31 00:12:41.912724 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.912693 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4"] Oct 31 00:12:41.913090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.913076 5073 cpu_manager.go:414] "containerMap: removing container" podUID="73d81019-1458-4b13-89bb-6967c062dac8" containerName="ta-container" Oct 31 00:12:41.913183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.913092 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d81019-1458-4b13-89bb-6967c062dac8" containerName="ta-container" Oct 31 00:12:41.913183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.913149 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d81019-1458-4b13-89bb-6967c062dac8" containerName="ta-container" Oct 31 00:12:41.916294 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.916277 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:41.929160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.929135 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4"] Oct 31 00:12:41.965006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.964975 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:41.965293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.965039 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqgn\" (UniqueName: \"kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:41.965293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:41.965168 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.066633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.066578 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.066633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.066628 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.066765 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.066656 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqgn\" (UniqueName: \"kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.067321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.067303 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.069102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.069086 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.078812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.078778 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqgn\" (UniqueName: \"kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn\") pod \"data-science-collector-targetallocator-599fd95c45-979k4\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.251015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.250987 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:42.525628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.525601 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4"] Oct 31 00:12:42.529155 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:42.529127 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d4ce320_1fc9_4aca_ad1e_f4aecf77110b.slice/crio-348e224518aa3dc52ff7bb6daaa98168f29450d09c68519af91cef6866a73b3b WatchSource:0}: Error finding container 348e224518aa3dc52ff7bb6daaa98168f29450d09c68519af91cef6866a73b3b: Status 404 returned error can't find the container with id 348e224518aa3dc52ff7bb6daaa98168f29450d09c68519af91cef6866a73b3b Oct 31 00:12:42.531231 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.531198 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:42.531306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:42.531254 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:43.084518 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:43.084483 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:12:43.093061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.092986 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" event={"ID":"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b","Type":"ContainerStarted","Data":"d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca"} Oct 31 00:12:43.093061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.093036 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" event={"ID":"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b","Type":"ContainerStarted","Data":"348e224518aa3dc52ff7bb6daaa98168f29450d09c68519af91cef6866a73b3b"} Oct 31 00:12:43.093381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.093355 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:43.095194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.095168 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:43.114206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.113992 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" podStartSLOduration=2.113975694 podStartE2EDuration="2.113975694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:43.110840666 +0000 UTC m=+847.171615364" watchObservedRunningTime="2026-10-31 00:12:43.113975694 +0000 UTC m=+847.174750392" Oct 31 00:12:43.165203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.165179 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7"] Oct 31 00:12:43.165368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.165351 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" podUID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" containerName="ta-container" containerID="cri-o://974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9" gracePeriod=30 Oct 31 00:12:43.419177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.419151 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:43.479771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.479739 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal\" (UniqueName: \"kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal\") pod \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " Oct 31 00:12:43.479909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.479851 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert\") pod \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " Oct 31 00:12:43.479909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.479874 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs\" (UniqueName: \"kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs\") pod \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\" (UID: \"c8a3d11c-1326-4033-b3d6-c7f8d0279069\") " Oct 31 00:12:43.480067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.480045 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal" pod "c8a3d11c-1326-4033-b3d6-c7f8d0279069" (UID: "c8a3d11c-1326-4033-b3d6-c7f8d0279069"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:43.482019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.481993 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert" pod "c8a3d11c-1326-4033-b3d6-c7f8d0279069" (UID: "c8a3d11c-1326-4033-b3d6-c7f8d0279069"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:43.482131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.482069 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs" pod "c8a3d11c-1326-4033-b3d6-c7f8d0279069" (UID: "c8a3d11c-1326-4033-b3d6-c7f8d0279069"). InnerVolumeSpecName "kube-api-access-d5mzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:43.580448 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.580424 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c8a3d11c-1326-4033-b3d6-c7f8d0279069-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:43.580448 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.580448 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c8a3d11c-1326-4033-b3d6-c7f8d0279069-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:43.580585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:43.580459 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5mzs\" (UniqueName: \"kubernetes.io/projected/c8a3d11c-1326-4033-b3d6-c7f8d0279069-kube-api-access-d5mzs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:44.100378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.100351 5073 generic.go:356] "Generic (PLEG): container finished" podID="47046010-bdeb-41ca-aa81-2a6b6445641e" containerID="d336e1fb20f6698b65379300caf9f4fd69a9d4c0b8e7d41cfc4f480f0ed7aab5" exitCode=0 Oct 31 00:12:44.100758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.100432 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"47046010-bdeb-41ca-aa81-2a6b6445641e","Type":"ContainerDied","Data":"d336e1fb20f6698b65379300caf9f4fd69a9d4c0b8e7d41cfc4f480f0ed7aab5"} Oct 31 00:12:44.102019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.102000 5073 generic.go:356] "Generic (PLEG): container finished" podID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" containerID="974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9" exitCode=0 Oct 31 00:12:44.102115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.102051 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" Oct 31 00:12:44.102115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.102072 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" event={"ID":"c8a3d11c-1326-4033-b3d6-c7f8d0279069","Type":"ContainerDied","Data":"974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9"} Oct 31 00:12:44.102208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.102129 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7" event={"ID":"c8a3d11c-1326-4033-b3d6-c7f8d0279069","Type":"ContainerDied","Data":"c2b65d33a3da00632c8ad5657f996cec0097d1037b294cc3ae9393c976774f40"} Oct 31 00:12:44.102208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.102154 5073 scope.go:122] "RemoveContainer" containerID="974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9" Oct 31 00:12:44.113694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.113609 5073 scope.go:122] "RemoveContainer" containerID="974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9" Oct 31 00:12:44.113936 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:44.113914 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9\": container with ID starting with 974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9 not found: ID does not exist" containerID="974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9" Oct 31 00:12:44.114020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.113942 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9"} err="failed to get container status \"974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9\": rpc error: code = NotFound desc = could not find container \"974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9\": container with ID starting with 974d8411c5560d8bae2f26498104de15c17168d2d351cbe19d29b88cc63a7ab9 not found: ID does not exist" Oct 31 00:12:44.120188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.120164 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7"] Oct 31 00:12:44.123768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.123746 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-c75f89845-6nzf7"] Oct 31 00:12:44.167619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.167599 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:44.185700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.185678 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert\") pod \"47046010-bdeb-41ca-aa81-2a6b6445641e\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " Oct 31 00:12:44.185824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.185716 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs\") pod \"47046010-bdeb-41ca-aa81-2a6b6445641e\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " Oct 31 00:12:44.185824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.185736 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal\") pod \"47046010-bdeb-41ca-aa81-2a6b6445641e\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " Oct 31 00:12:44.185943 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.185883 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs\") pod \"47046010-bdeb-41ca-aa81-2a6b6445641e\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " Oct 31 00:12:44.185943 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.185929 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs\") pod \"47046010-bdeb-41ca-aa81-2a6b6445641e\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " Oct 31 00:12:44.186047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.185982 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp\" (UniqueName: \"kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp\") pod \"47046010-bdeb-41ca-aa81-2a6b6445641e\" (UID: \"47046010-bdeb-41ca-aa81-2a6b6445641e\") " Oct 31 00:12:44.186178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.186146 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal" pod "47046010-bdeb-41ca-aa81-2a6b6445641e" (UID: "47046010-bdeb-41ca-aa81-2a6b6445641e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:44.186387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.186365 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/47046010-bdeb-41ca-aa81-2a6b6445641e-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:44.188814 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.188626 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs" pod "47046010-bdeb-41ca-aa81-2a6b6445641e" (UID: "47046010-bdeb-41ca-aa81-2a6b6445641e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:44.189185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.189166 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs" pod "47046010-bdeb-41ca-aa81-2a6b6445641e" (UID: "47046010-bdeb-41ca-aa81-2a6b6445641e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:44.189290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.189243 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp" pod "47046010-bdeb-41ca-aa81-2a6b6445641e" (UID: "47046010-bdeb-41ca-aa81-2a6b6445641e"). InnerVolumeSpecName "kube-api-access-47jfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:44.190316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.190037 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs" pod "47046010-bdeb-41ca-aa81-2a6b6445641e" (UID: "47046010-bdeb-41ca-aa81-2a6b6445641e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:44.190316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.190059 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert" pod "47046010-bdeb-41ca-aa81-2a6b6445641e" (UID: "47046010-bdeb-41ca-aa81-2a6b6445641e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:44.287823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.287725 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:44.287823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.287751 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:44.287823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.287765 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jfp\" (UniqueName: \"kubernetes.io/projected/47046010-bdeb-41ca-aa81-2a6b6445641e-kube-api-access-47jfp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:44.287823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.287807 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:44.287823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:44.287822 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47046010-bdeb-41ca-aa81-2a6b6445641e-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:45.109315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.109284 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.109928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.109324 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"47046010-bdeb-41ca-aa81-2a6b6445641e","Type":"ContainerDied","Data":"592660b41df978ac687b0566d86f688cce7236244b96468bd18ac8fb03e03631"} Oct 31 00:12:45.109928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.109386 5073 scope.go:122] "RemoveContainer" containerID="d336e1fb20f6698b65379300caf9f4fd69a9d4c0b8e7d41cfc4f480f0ed7aab5" Oct 31 00:12:45.130517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.130491 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:45.135522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.135501 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:45.162000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.161976 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:45.162344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.162330 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" containerName="ta-container" Oct 31 00:12:45.162389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.162346 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" containerName="ta-container" Oct 31 00:12:45.162389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.162360 5073 cpu_manager.go:414] "containerMap: removing container" podUID="47046010-bdeb-41ca-aa81-2a6b6445641e" containerName="otc-container" Oct 31 00:12:45.162389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.162365 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47046010-bdeb-41ca-aa81-2a6b6445641e" containerName="otc-container" Oct 31 00:12:45.162509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.162447 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" containerName="ta-container" Oct 31 00:12:45.162509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.162457 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="47046010-bdeb-41ca-aa81-2a6b6445641e" containerName="otc-container" Oct 31 00:12:45.166929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.166913 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.171045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.171019 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:45.171045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.171030 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:45.173934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.173876 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:45.175575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.175558 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:12:45.176907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.176887 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:45.193712 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.193692 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.193856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.193737 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.193856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.193757 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.194007 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.193981 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrl4\" (UniqueName: \"kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.194084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.194064 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.194155 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.194119 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.271957 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.271935 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:45.272051 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.271967 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:45.295386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.295363 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.295465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.295392 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.295465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.295420 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrl4\" (UniqueName: \"kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.295465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.295450 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.295621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.295471 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.295621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.295523 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.296422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.296387 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.298018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.297994 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.298152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.298134 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.298193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.298160 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.298230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.298213 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.308165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.308148 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrl4\" (UniqueName: \"kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4\") pod \"data-science-collector-collector-1\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.480462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.480411 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:45.756162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.756132 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:45.758745 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:45.758718 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12917cb_20d2_4012_b000_b500160dba29.slice/crio-c9383e7c8b69235365cc2720b18e95dedd6ff0c75a6516c93c00598a06c4f053 WatchSource:0}: Error finding container c9383e7c8b69235365cc2720b18e95dedd6ff0c75a6516c93c00598a06c4f053: Status 404 returned error can't find the container with id c9383e7c8b69235365cc2720b18e95dedd6ff0c75a6516c93c00598a06c4f053 Oct 31 00:12:45.761640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.761611 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:45.761720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:45.761670 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:46.060581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:46.060509 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47046010-bdeb-41ca-aa81-2a6b6445641e" path="/var/lib/kubelet/pods/47046010-bdeb-41ca-aa81-2a6b6445641e/volumes" Oct 31 00:12:46.060912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:46.060898 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a3d11c-1326-4033-b3d6-c7f8d0279069" path="/var/lib/kubelet/pods/c8a3d11c-1326-4033-b3d6-c7f8d0279069/volumes" Oct 31 00:12:46.114624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:46.114594 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f12917cb-20d2-4012-b000-b500160dba29","Type":"ContainerStarted","Data":"1ed68c313780344baaabf9c7607787a9b54db15fa9fc2a309b7d6df1ac94e2a3"} Oct 31 00:12:46.115013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:46.114634 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f12917cb-20d2-4012-b000-b500160dba29","Type":"ContainerStarted","Data":"c9383e7c8b69235365cc2720b18e95dedd6ff0c75a6516c93c00598a06c4f053"} Oct 31 00:12:46.135619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:46.135572 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.135554249 podStartE2EDuration="1.135554249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:46.135119589 +0000 UTC m=+850.195894281" watchObservedRunningTime="2026-10-31 00:12:46.135554249 +0000 UTC m=+850.196328937" Oct 31 00:12:53.980285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:53.980251 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:53.980682 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:53.980474 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f12917cb-20d2-4012-b000-b500160dba29" containerName="otc-container" containerID="cri-o://1ed68c313780344baaabf9c7607787a9b54db15fa9fc2a309b7d6df1ac94e2a3" gracePeriod=30 Oct 31 00:12:54.145395 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.145365 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl"] Oct 31 00:12:54.148754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.148728 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.165841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.165034 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl"] Oct 31 00:12:54.168072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.168052 5073 generic.go:356] "Generic (PLEG): container finished" podID="f12917cb-20d2-4012-b000-b500160dba29" containerID="1ed68c313780344baaabf9c7607787a9b54db15fa9fc2a309b7d6df1ac94e2a3" exitCode=0 Oct 31 00:12:54.168203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.168186 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f12917cb-20d2-4012-b000-b500160dba29","Type":"ContainerDied","Data":"1ed68c313780344baaabf9c7607787a9b54db15fa9fc2a309b7d6df1ac94e2a3"} Oct 31 00:12:54.226220 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.226197 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:54.267609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.267579 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs\") pod \"f12917cb-20d2-4012-b000-b500160dba29\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " Oct 31 00:12:54.267754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.267625 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal\") pod \"f12917cb-20d2-4012-b000-b500160dba29\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " Oct 31 00:12:54.267754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.267646 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4\" (UniqueName: \"kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4\") pod \"f12917cb-20d2-4012-b000-b500160dba29\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " Oct 31 00:12:54.267754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.267664 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs\") pod \"f12917cb-20d2-4012-b000-b500160dba29\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " Oct 31 00:12:54.267754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.267693 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert\") pod \"f12917cb-20d2-4012-b000-b500160dba29\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " Oct 31 00:12:54.268000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.267826 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs\") pod \"f12917cb-20d2-4012-b000-b500160dba29\" (UID: \"f12917cb-20d2-4012-b000-b500160dba29\") " Oct 31 00:12:54.268059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.268040 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9566\" (UniqueName: \"kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.268146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.268124 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.268208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.268149 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal" pod "f12917cb-20d2-4012-b000-b500160dba29" (UID: "f12917cb-20d2-4012-b000-b500160dba29"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:54.268316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.268292 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.268438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.268419 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f12917cb-20d2-4012-b000-b500160dba29-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:54.270741 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.270701 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert" pod "f12917cb-20d2-4012-b000-b500160dba29" (UID: "f12917cb-20d2-4012-b000-b500160dba29"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:54.271417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.271388 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4" pod "f12917cb-20d2-4012-b000-b500160dba29" (UID: "f12917cb-20d2-4012-b000-b500160dba29"). InnerVolumeSpecName "kube-api-access-2xrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:54.271531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.271437 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs" pod "f12917cb-20d2-4012-b000-b500160dba29" (UID: "f12917cb-20d2-4012-b000-b500160dba29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:54.271531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.271456 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs" pod "f12917cb-20d2-4012-b000-b500160dba29" (UID: "f12917cb-20d2-4012-b000-b500160dba29"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:54.271531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.271470 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs" pod "f12917cb-20d2-4012-b000-b500160dba29" (UID: "f12917cb-20d2-4012-b000-b500160dba29"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:54.369643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369614 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.369762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369673 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.369762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369712 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9566\" (UniqueName: \"kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.369886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369759 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:54.369886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369776 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xrl4\" (UniqueName: \"kubernetes.io/projected/f12917cb-20d2-4012-b000-b500160dba29-kube-api-access-2xrl4\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:54.369886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369815 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:54.369886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369830 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:54.369886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.369843 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f12917cb-20d2-4012-b000-b500160dba29-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:54.370387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.370314 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.372127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.372107 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.382608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.382587 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9566\" (UniqueName: \"kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566\") pod \"data-science-collector-targetallocator-5b798b684-crvwl\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.468609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.468585 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:54.756410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.756376 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl"] Oct 31 00:12:54.759236 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:54.759209 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19822666_aebb_4df2_8d8b_878cecf60221.slice/crio-8f3c282290e4a6cede8aa2dc80a2360068b9851699c45d22b6e2745089f1dd2b WatchSource:0}: Error finding container 8f3c282290e4a6cede8aa2dc80a2360068b9851699c45d22b6e2745089f1dd2b: Status 404 returned error can't find the container with id 8f3c282290e4a6cede8aa2dc80a2360068b9851699c45d22b6e2745089f1dd2b Oct 31 00:12:54.761326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.761295 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:54.761409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:54.761352 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:55.174419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.174343 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" event={"ID":"19822666-aebb-4df2-8d8b-878cecf60221","Type":"ContainerStarted","Data":"b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a"} Oct 31 00:12:55.174419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.174383 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" event={"ID":"19822666-aebb-4df2-8d8b-878cecf60221","Type":"ContainerStarted","Data":"8f3c282290e4a6cede8aa2dc80a2360068b9851699c45d22b6e2745089f1dd2b"} Oct 31 00:12:55.174920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.174529 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:55.175950 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.175930 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.176108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.175925 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f12917cb-20d2-4012-b000-b500160dba29","Type":"ContainerDied","Data":"c9383e7c8b69235365cc2720b18e95dedd6ff0c75a6516c93c00598a06c4f053"} Oct 31 00:12:55.176108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.176086 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:12:55.176108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.176104 5073 scope.go:122] "RemoveContainer" containerID="1ed68c313780344baaabf9c7607787a9b54db15fa9fc2a309b7d6df1ac94e2a3" Oct 31 00:12:55.194289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.194250 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" podStartSLOduration=1.194231039 podStartE2EDuration="1.194231039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:55.192429162 +0000 UTC m=+859.253203860" watchObservedRunningTime="2026-10-31 00:12:55.194231039 +0000 UTC m=+859.255005733" Oct 31 00:12:55.206356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.206333 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:55.210940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.210920 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:55.239952 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.239926 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:55.240459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.240435 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f12917cb-20d2-4012-b000-b500160dba29" containerName="otc-container" Oct 31 00:12:55.240459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.240459 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12917cb-20d2-4012-b000-b500160dba29" containerName="otc-container" Oct 31 00:12:55.240599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.240589 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12917cb-20d2-4012-b000-b500160dba29" containerName="otc-container" Oct 31 00:12:55.245074 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.245051 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.252178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.252154 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:55.252300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.252238 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:55.252370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.252301 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:55.252426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.252370 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:55.254303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.254286 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:55.254860 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.254842 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:12:55.259730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.259709 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:55.262653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.262634 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4"] Oct 31 00:12:55.262851 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.262827 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" podUID="1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" containerName="ta-container" containerID="cri-o://d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca" gracePeriod=30 Oct 31 00:12:55.277378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.276180 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.277378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.276224 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.277378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.276286 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.277378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.276334 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9q68\" (UniqueName: \"kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.277378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.276391 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.277378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.276487 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.376917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.376892 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.377070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.376923 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.377070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.376947 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.377070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.376985 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q68\" (UniqueName: \"kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.377070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.377019 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.377272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.377086 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.377690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.377604 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.379583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.379561 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.379682 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.379561 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.379682 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.379595 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.379752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.379738 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.391697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.391670 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q68\" (UniqueName: \"kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68\") pod \"data-science-collector-collector-1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.494503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.494481 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:55.561727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.561701 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:12:55.578704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.578681 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn\" (UniqueName: \"kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn\") pod \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " Oct 31 00:12:55.578848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.578716 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal\" (UniqueName: \"kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal\") pod \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " Oct 31 00:12:55.578904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.578882 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert\") pod \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\" (UID: \"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b\") " Oct 31 00:12:55.579115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.579089 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal" pod "1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" (UID: "1d4ce320-1fc9-4aca-ad1e-f4aecf77110b"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:55.579241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.579165 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.580874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.580843 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn" pod "1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" (UID: "1d4ce320-1fc9-4aca-ad1e-f4aecf77110b"). InnerVolumeSpecName "kube-api-access-zqqgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:55.581087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.581060 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert" pod "1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" (UID: "1d4ce320-1fc9-4aca-ad1e-f4aecf77110b"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:55.680612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.680573 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.680612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.680603 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqgn\" (UniqueName: \"kubernetes.io/projected/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b-kube-api-access-zqqgn\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.840930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.840904 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:12:55.844971 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:12:55.844940 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244127b9_81f2_431c_a30e_7fdfd89de3b1.slice/crio-4086fd9b1b69cffe55f230290c01813966df614e074b15ad40990d54bc532689 WatchSource:0}: Error finding container 4086fd9b1b69cffe55f230290c01813966df614e074b15ad40990d54bc532689: Status 404 returned error can't find the container with id 4086fd9b1b69cffe55f230290c01813966df614e074b15ad40990d54bc532689 Oct 31 00:12:55.847116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.847085 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:55.847185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:55.847143 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:12:56.060230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.060186 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12917cb-20d2-4012-b000-b500160dba29" path="/var/lib/kubelet/pods/f12917cb-20d2-4012-b000-b500160dba29/volumes" Oct 31 00:12:56.182911 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.182819 5073 generic.go:356] "Generic (PLEG): container finished" podID="1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" containerID="d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca" exitCode=0 Oct 31 00:12:56.182911 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.182873 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" Oct 31 00:12:56.182911 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.182897 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" event={"ID":"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b","Type":"ContainerDied","Data":"d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca"} Oct 31 00:12:56.183405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.182929 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4" event={"ID":"1d4ce320-1fc9-4aca-ad1e-f4aecf77110b","Type":"ContainerDied","Data":"348e224518aa3dc52ff7bb6daaa98168f29450d09c68519af91cef6866a73b3b"} Oct 31 00:12:56.183405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.182946 5073 scope.go:122] "RemoveContainer" containerID="d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca" Oct 31 00:12:56.185219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.185135 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"244127b9-81f2-431c-a30e-7fdfd89de3b1","Type":"ContainerStarted","Data":"5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880"} Oct 31 00:12:56.185219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.185184 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"244127b9-81f2-431c-a30e-7fdfd89de3b1","Type":"ContainerStarted","Data":"4086fd9b1b69cffe55f230290c01813966df614e074b15ad40990d54bc532689"} Oct 31 00:12:56.191619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.191604 5073 scope.go:122] "RemoveContainer" containerID="d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca" Oct 31 00:12:56.191912 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:12:56.191893 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca\": container with ID starting with d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca not found: ID does not exist" containerID="d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca" Oct 31 00:12:56.191983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.191923 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca"} err="failed to get container status \"d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca\": rpc error: code = NotFound desc = could not find container \"d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca\": container with ID starting with d0c74b5572bc8eb150b96342ce45710eceffdb41e1e9dd497c1d5244e5a164ca not found: ID does not exist" Oct 31 00:12:56.200774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.200753 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4"] Oct 31 00:12:56.205021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.205001 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599fd95c45-979k4"] Oct 31 00:12:56.223883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:56.223830 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.223816338 podStartE2EDuration="1.223816338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:56.223503435 +0000 UTC m=+860.284278132" watchObservedRunningTime="2026-10-31 00:12:56.223816338 +0000 UTC m=+860.284591033" Oct 31 00:12:58.058705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:12:58.058667 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" path="/var/lib/kubelet/pods/1d4ce320-1fc9-4aca-ad1e-f4aecf77110b/volumes" Oct 31 00:13:06.389254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.389223 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:06.389901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.389404 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="244127b9-81f2-431c-a30e-7fdfd89de3b1" containerName="otc-container" containerID="cri-o://5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880" gracePeriod=30 Oct 31 00:13:06.504118 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.504088 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj"] Oct 31 00:13:06.504440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.504428 5073 cpu_manager.go:414] "containerMap: removing container" podUID="1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" containerName="ta-container" Oct 31 00:13:06.504482 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.504442 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" containerName="ta-container" Oct 31 00:13:06.504538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.504529 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4ce320-1fc9-4aca-ad1e-f4aecf77110b" containerName="ta-container" Oct 31 00:13:06.507686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.507655 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.517321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.517296 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj"] Oct 31 00:13:06.570537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.570494 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmpr\" (UniqueName: \"kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.570696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.570556 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.570696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.570589 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.646613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.646558 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:06.671855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.671827 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.672001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.671957 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmpr\" (UniqueName: \"kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.672001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.671983 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.672620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.672591 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.674711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.674669 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.687216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.687195 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmpr\" (UniqueName: \"kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr\") pod \"data-science-collector-targetallocator-55b898d755-wbghj\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.772893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.772862 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68\" (UniqueName: \"kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68\") pod \"244127b9-81f2-431c-a30e-7fdfd89de3b1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " Oct 31 00:13:06.773039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.772915 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal\") pod \"244127b9-81f2-431c-a30e-7fdfd89de3b1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " Oct 31 00:13:06.773039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.772947 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs\") pod \"244127b9-81f2-431c-a30e-7fdfd89de3b1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " Oct 31 00:13:06.773039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.773021 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs\") pod \"244127b9-81f2-431c-a30e-7fdfd89de3b1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " Oct 31 00:13:06.773208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.773073 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs\") pod \"244127b9-81f2-431c-a30e-7fdfd89de3b1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " Oct 31 00:13:06.773208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.773139 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert\") pod \"244127b9-81f2-431c-a30e-7fdfd89de3b1\" (UID: \"244127b9-81f2-431c-a30e-7fdfd89de3b1\") " Oct 31 00:13:06.773391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.773358 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal" pod "244127b9-81f2-431c-a30e-7fdfd89de3b1" (UID: "244127b9-81f2-431c-a30e-7fdfd89de3b1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:06.775162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.775134 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68" pod "244127b9-81f2-431c-a30e-7fdfd89de3b1" (UID: "244127b9-81f2-431c-a30e-7fdfd89de3b1"). InnerVolumeSpecName "kube-api-access-c9q68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:06.775321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.775225 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs" pod "244127b9-81f2-431c-a30e-7fdfd89de3b1" (UID: "244127b9-81f2-431c-a30e-7fdfd89de3b1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:06.775321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.775302 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert" pod "244127b9-81f2-431c-a30e-7fdfd89de3b1" (UID: "244127b9-81f2-431c-a30e-7fdfd89de3b1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:06.775734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.775709 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs" pod "244127b9-81f2-431c-a30e-7fdfd89de3b1" (UID: "244127b9-81f2-431c-a30e-7fdfd89de3b1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:06.775734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.775725 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs" pod "244127b9-81f2-431c-a30e-7fdfd89de3b1" (UID: "244127b9-81f2-431c-a30e-7fdfd89de3b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:06.820934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.820899 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:06.879806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.878714 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:06.879806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.878751 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:06.879806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.878768 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:06.879806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.879096 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9q68\" (UniqueName: \"kubernetes.io/projected/244127b9-81f2-431c-a30e-7fdfd89de3b1-kube-api-access-c9q68\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:06.879806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.879123 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/244127b9-81f2-431c-a30e-7fdfd89de3b1-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:06.879806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:06.879140 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244127b9-81f2-431c-a30e-7fdfd89de3b1-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:07.097636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.097487 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj"] Oct 31 00:13:07.100496 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:07.100472 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57c71310_9786_4924_9db1_09f59ba182dc.slice/crio-ae780bd680d0b496e79d2b638bde56bae50a92dbe1dc8107421191874fd00b2f WatchSource:0}: Error finding container ae780bd680d0b496e79d2b638bde56bae50a92dbe1dc8107421191874fd00b2f: Status 404 returned error can't find the container with id ae780bd680d0b496e79d2b638bde56bae50a92dbe1dc8107421191874fd00b2f Oct 31 00:13:07.102462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.102434 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:07.102523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.102485 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:07.237080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.237041 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" event={"ID":"57c71310-9786-4924-9db1-09f59ba182dc","Type":"ContainerStarted","Data":"8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea"} Oct 31 00:13:07.237291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.237097 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:07.237291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.237115 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" event={"ID":"57c71310-9786-4924-9db1-09f59ba182dc","Type":"ContainerStarted","Data":"ae780bd680d0b496e79d2b638bde56bae50a92dbe1dc8107421191874fd00b2f"} Oct 31 00:13:07.238457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.238433 5073 generic.go:356] "Generic (PLEG): container finished" podID="244127b9-81f2-431c-a30e-7fdfd89de3b1" containerID="5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880" exitCode=0 Oct 31 00:13:07.238577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.238512 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"244127b9-81f2-431c-a30e-7fdfd89de3b1","Type":"ContainerDied","Data":"5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880"} Oct 31 00:13:07.238577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.238544 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"244127b9-81f2-431c-a30e-7fdfd89de3b1","Type":"ContainerDied","Data":"4086fd9b1b69cffe55f230290c01813966df614e074b15ad40990d54bc532689"} Oct 31 00:13:07.238577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.238565 5073 scope.go:122] "RemoveContainer" containerID="5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880" Oct 31 00:13:07.238676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.238655 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.239939 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.239898 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" podUID="57c71310-9786-4924-9db1-09f59ba182dc" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.71:8080/readyz\": dial tcp 10.132.0.71:8080: connect: connection refused" Oct 31 00:13:07.249577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.249559 5073 scope.go:122] "RemoveContainer" containerID="5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880" Oct 31 00:13:07.249902 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:07.249881 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880\": container with ID starting with 5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880 not found: ID does not exist" containerID="5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880" Oct 31 00:13:07.249965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.249911 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880"} err="failed to get container status \"5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880\": rpc error: code = NotFound desc = could not find container \"5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880\": container with ID starting with 5f8f44074e3531b7a6fb21cf21642679413b2f25cbb513d695cc026980ccd880 not found: ID does not exist" Oct 31 00:13:07.258865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.258817 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" podStartSLOduration=1.258801015 podStartE2EDuration="1.258801015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:07.257954879 +0000 UTC m=+871.318729573" watchObservedRunningTime="2026-10-31 00:13:07.258801015 +0000 UTC m=+871.319575706" Oct 31 00:13:07.273413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.273391 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:07.278931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.278910 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:07.305643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.305608 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:07.306432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.306408 5073 cpu_manager.go:414] "containerMap: removing container" podUID="244127b9-81f2-431c-a30e-7fdfd89de3b1" containerName="otc-container" Oct 31 00:13:07.306514 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.306434 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244127b9-81f2-431c-a30e-7fdfd89de3b1" containerName="otc-container" Oct 31 00:13:07.306679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.306665 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="244127b9-81f2-431c-a30e-7fdfd89de3b1" containerName="otc-container" Oct 31 00:13:07.312070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.312050 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.316384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.316364 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:07.316705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.316663 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\"" Oct 31 00:13:07.316867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.316732 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:07.316867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.316818 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:07.316867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.316864 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:07.317043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.316905 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:07.317656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.317634 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:07.384193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.384136 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.384307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.384253 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.384307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.384288 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.384430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.384412 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhdv\" (UniqueName: \"kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.384473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.384444 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.384557 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.384542 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.485852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.485823 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.486206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.485869 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.486206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.485912 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.486206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.485931 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.486206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.485957 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhdv\" (UniqueName: \"kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.486206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.485976 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.486673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.486649 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.488764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.488740 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.488897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.488874 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.489038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.489019 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.489673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.489652 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.498090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.498068 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhdv\" (UniqueName: \"kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv\") pod \"data-science-collector-collector-1\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.626485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.626461 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:07.901208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.901131 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:07.904189 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:07.904160 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562a3e76_b186_4b7b_9f69_ea57915ec91a.slice/crio-9efb92c09aba6cfc68e31ed9b054e3bd002c2ded62d70728a41f00defbde2a99 WatchSource:0}: Error finding container 9efb92c09aba6cfc68e31ed9b054e3bd002c2ded62d70728a41f00defbde2a99: Status 404 returned error can't find the container with id 9efb92c09aba6cfc68e31ed9b054e3bd002c2ded62d70728a41f00defbde2a99 Oct 31 00:13:07.906343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.906308 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:07.906415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:07.906369 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:08.058980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.058948 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244127b9-81f2-431c-a30e-7fdfd89de3b1" path="/var/lib/kubelet/pods/244127b9-81f2-431c-a30e-7fdfd89de3b1/volumes" Oct 31 00:13:08.245314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.245278 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"562a3e76-b186-4b7b-9f69-ea57915ec91a","Type":"ContainerStarted","Data":"7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b"} Oct 31 00:13:08.245482 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.245335 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"562a3e76-b186-4b7b-9f69-ea57915ec91a","Type":"ContainerStarted","Data":"9efb92c09aba6cfc68e31ed9b054e3bd002c2ded62d70728a41f00defbde2a99"} Oct 31 00:13:08.247372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.247353 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:08.266231 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.266176 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.2661636600000001 podStartE2EDuration="1.26616366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:08.265070062 +0000 UTC m=+872.325844759" watchObservedRunningTime="2026-10-31 00:13:08.26616366 +0000 UTC m=+872.326938378" Oct 31 00:13:08.314740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.314702 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl"] Oct 31 00:13:08.314928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.314901 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" podUID="19822666-aebb-4df2-8d8b-878cecf60221" containerName="ta-container" containerID="cri-o://b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a" gracePeriod=30 Oct 31 00:13:08.558820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.558772 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:13:08.697603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.697574 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566\" (UniqueName: \"kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566\") pod \"19822666-aebb-4df2-8d8b-878cecf60221\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " Oct 31 00:13:08.697766 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.697627 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert\") pod \"19822666-aebb-4df2-8d8b-878cecf60221\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " Oct 31 00:13:08.697766 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.697674 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal\" (UniqueName: \"kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal\") pod \"19822666-aebb-4df2-8d8b-878cecf60221\" (UID: \"19822666-aebb-4df2-8d8b-878cecf60221\") " Oct 31 00:13:08.698105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.698082 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal" pod "19822666-aebb-4df2-8d8b-878cecf60221" (UID: "19822666-aebb-4df2-8d8b-878cecf60221"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:08.699887 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.699862 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert" pod "19822666-aebb-4df2-8d8b-878cecf60221" (UID: "19822666-aebb-4df2-8d8b-878cecf60221"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:08.699974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.699908 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566" pod "19822666-aebb-4df2-8d8b-878cecf60221" (UID: "19822666-aebb-4df2-8d8b-878cecf60221"). InnerVolumeSpecName "kube-api-access-v9566". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:08.798476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.798412 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/19822666-aebb-4df2-8d8b-878cecf60221-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:08.798476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.798444 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/19822666-aebb-4df2-8d8b-878cecf60221-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:08.798476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:08.798454 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9566\" (UniqueName: \"kubernetes.io/projected/19822666-aebb-4df2-8d8b-878cecf60221-kube-api-access-v9566\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.252384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.252341 5073 generic.go:356] "Generic (PLEG): container finished" podID="19822666-aebb-4df2-8d8b-878cecf60221" containerID="b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a" exitCode=0 Oct 31 00:13:09.252540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.252388 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" Oct 31 00:13:09.252540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.252427 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" event={"ID":"19822666-aebb-4df2-8d8b-878cecf60221","Type":"ContainerDied","Data":"b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a"} Oct 31 00:13:09.252540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.252469 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl" event={"ID":"19822666-aebb-4df2-8d8b-878cecf60221","Type":"ContainerDied","Data":"8f3c282290e4a6cede8aa2dc80a2360068b9851699c45d22b6e2745089f1dd2b"} Oct 31 00:13:09.252540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.252493 5073 scope.go:122] "RemoveContainer" containerID="b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a" Oct 31 00:13:09.261524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.261506 5073 scope.go:122] "RemoveContainer" containerID="b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a" Oct 31 00:13:09.261776 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:09.261758 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a\": container with ID starting with b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a not found: ID does not exist" containerID="b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a" Oct 31 00:13:09.261864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.261801 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a"} err="failed to get container status \"b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a\": rpc error: code = NotFound desc = could not find container \"b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a\": container with ID starting with b21ca8be7f5aa362a6a3afc55feae1ff8bc6022b5a881007b7d452bddfce725a not found: ID does not exist" Oct 31 00:13:09.272998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.272968 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl"] Oct 31 00:13:09.277389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:09.277366 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b798b684-crvwl"] Oct 31 00:13:10.058924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:10.058893 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19822666-aebb-4df2-8d8b-878cecf60221" path="/var/lib/kubelet/pods/19822666-aebb-4df2-8d8b-878cecf60221/volumes" Oct 31 00:13:13.085901 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:13.085865 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:13:18.725672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.725631 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:18.726179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.725954 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="562a3e76-b186-4b7b-9f69-ea57915ec91a" containerName="otc-container" containerID="cri-o://7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b" gracePeriod=30 Oct 31 00:13:18.824905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.824867 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw"] Oct 31 00:13:18.825291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.825273 5073 cpu_manager.go:414] "containerMap: removing container" podUID="19822666-aebb-4df2-8d8b-878cecf60221" containerName="ta-container" Oct 31 00:13:18.825365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.825295 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19822666-aebb-4df2-8d8b-878cecf60221" containerName="ta-container" Oct 31 00:13:18.825449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.825436 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="19822666-aebb-4df2-8d8b-878cecf60221" containerName="ta-container" Oct 31 00:13:18.830027 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.830005 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:18.840873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.840851 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw"] Oct 31 00:13:18.983883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.983855 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:18.984017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.983901 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:18.984073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:18.984029 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87p7z\" (UniqueName: \"kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.084331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.084303 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.084461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.084341 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.084461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.084369 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87p7z\" (UniqueName: \"kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.084970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.084950 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.087092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.087070 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.096732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.096707 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87p7z\" (UniqueName: \"kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z\") pod \"data-science-collector-targetallocator-6fd9967d54-d2nfw\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.141825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.141808 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:19.416234 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.416155 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw"] Oct 31 00:13:19.419064 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:19.419035 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93d53244_bc34_4bb0_8e9c_225e250753c3.slice/crio-4646e07a1535a8f1c8adde5ecda212974b69af187f031820f20c25f95d685417 WatchSource:0}: Error finding container 4646e07a1535a8f1c8adde5ecda212974b69af187f031820f20c25f95d685417: Status 404 returned error can't find the container with id 4646e07a1535a8f1c8adde5ecda212974b69af187f031820f20c25f95d685417 Oct 31 00:13:19.421325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.421290 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:19.421395 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.421348 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:19.968267 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:19.968245 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.094229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094142 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal\") pod \"562a3e76-b186-4b7b-9f69-ea57915ec91a\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " Oct 31 00:13:20.094344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094228 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs\") pod \"562a3e76-b186-4b7b-9f69-ea57915ec91a\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " Oct 31 00:13:20.094344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094299 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv\" (UniqueName: \"kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv\") pod \"562a3e76-b186-4b7b-9f69-ea57915ec91a\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " Oct 31 00:13:20.094455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094347 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert\") pod \"562a3e76-b186-4b7b-9f69-ea57915ec91a\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " Oct 31 00:13:20.094455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094381 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs\") pod \"562a3e76-b186-4b7b-9f69-ea57915ec91a\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " Oct 31 00:13:20.094455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094407 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs\") pod \"562a3e76-b186-4b7b-9f69-ea57915ec91a\" (UID: \"562a3e76-b186-4b7b-9f69-ea57915ec91a\") " Oct 31 00:13:20.094665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094558 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal" pod "562a3e76-b186-4b7b-9f69-ea57915ec91a" (UID: "562a3e76-b186-4b7b-9f69-ea57915ec91a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:20.094665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.094657 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/562a3e76-b186-4b7b-9f69-ea57915ec91a-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.097101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.097069 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs" pod "562a3e76-b186-4b7b-9f69-ea57915ec91a" (UID: "562a3e76-b186-4b7b-9f69-ea57915ec91a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:20.097210 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.097103 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv" pod "562a3e76-b186-4b7b-9f69-ea57915ec91a" (UID: "562a3e76-b186-4b7b-9f69-ea57915ec91a"). InnerVolumeSpecName "kube-api-access-blhdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:20.097274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.097228 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert" pod "562a3e76-b186-4b7b-9f69-ea57915ec91a" (UID: "562a3e76-b186-4b7b-9f69-ea57915ec91a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:20.097274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.097250 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs" pod "562a3e76-b186-4b7b-9f69-ea57915ec91a" (UID: "562a3e76-b186-4b7b-9f69-ea57915ec91a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:20.097345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.097320 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs" pod "562a3e76-b186-4b7b-9f69-ea57915ec91a" (UID: "562a3e76-b186-4b7b-9f69-ea57915ec91a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:20.195063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.195034 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.195063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.195060 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhdv\" (UniqueName: \"kubernetes.io/projected/562a3e76-b186-4b7b-9f69-ea57915ec91a-kube-api-access-blhdv\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.195186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.195073 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.195186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.195083 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.195186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.195092 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562a3e76-b186-4b7b-9f69-ea57915ec91a-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.310357 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.310332 5073 generic.go:356] "Generic (PLEG): container finished" podID="562a3e76-b186-4b7b-9f69-ea57915ec91a" containerID="7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b" exitCode=0 Oct 31 00:13:20.310469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.310383 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.310469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.310422 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"562a3e76-b186-4b7b-9f69-ea57915ec91a","Type":"ContainerDied","Data":"7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b"} Oct 31 00:13:20.310556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.310491 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"562a3e76-b186-4b7b-9f69-ea57915ec91a","Type":"ContainerDied","Data":"9efb92c09aba6cfc68e31ed9b054e3bd002c2ded62d70728a41f00defbde2a99"} Oct 31 00:13:20.310556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.310524 5073 scope.go:122] "RemoveContainer" containerID="7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b" Oct 31 00:13:20.312262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.312239 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" event={"ID":"93d53244-bc34-4bb0-8e9c-225e250753c3","Type":"ContainerStarted","Data":"d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1"} Oct 31 00:13:20.312368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.312270 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" event={"ID":"93d53244-bc34-4bb0-8e9c-225e250753c3","Type":"ContainerStarted","Data":"4646e07a1535a8f1c8adde5ecda212974b69af187f031820f20c25f95d685417"} Oct 31 00:13:20.312368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.312353 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:20.314413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.314394 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:20.319769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.319753 5073 scope.go:122] "RemoveContainer" containerID="7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b" Oct 31 00:13:20.320046 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:20.320027 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b\": container with ID starting with 7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b not found: ID does not exist" containerID="7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b" Oct 31 00:13:20.320111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.320054 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b"} err="failed to get container status \"7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b\": rpc error: code = NotFound desc = could not find container \"7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b\": container with ID starting with 7164f7d261b25682e9d280ca3a8e6b9af864d437dc528c97b6e9aac213df231b not found: ID does not exist" Oct 31 00:13:20.333226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.333183 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" podStartSLOduration=2.333168183 podStartE2EDuration="2.333168183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:20.332187258 +0000 UTC m=+884.392961953" watchObservedRunningTime="2026-10-31 00:13:20.333168183 +0000 UTC m=+884.393942859" Oct 31 00:13:20.348272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.348202 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:20.351901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.351883 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:20.362636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.362617 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj"] Oct 31 00:13:20.362867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.362844 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" podUID="57c71310-9786-4924-9db1-09f59ba182dc" containerName="ta-container" containerID="cri-o://8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea" gracePeriod=30 Oct 31 00:13:20.379047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.379019 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:20.379603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.379583 5073 cpu_manager.go:414] "containerMap: removing container" podUID="562a3e76-b186-4b7b-9f69-ea57915ec91a" containerName="otc-container" Oct 31 00:13:20.379758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.379605 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562a3e76-b186-4b7b-9f69-ea57915ec91a" containerName="otc-container" Oct 31 00:13:20.379853 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.379758 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="562a3e76-b186-4b7b-9f69-ea57915ec91a" containerName="otc-container" Oct 31 00:13:20.382751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.382735 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.388004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.387983 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:20.390715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.390651 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:20.390954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.390739 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:20.390954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.390932 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:20.391636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.391582 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:13:20.391731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.391713 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:20.392974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.392951 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:20.498681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.498638 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.498681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.498677 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.498918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.498693 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.498918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.498835 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kff8\" (UniqueName: \"kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.498918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.498873 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.498918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.498900 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.599518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.599463 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.599518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.599506 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.599685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.599523 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.599685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.599541 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kff8\" (UniqueName: \"kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.599685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.599572 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.599685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.599597 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.600527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.600495 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.602299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.602276 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.602413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.602285 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.602495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.602476 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.603081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.603059 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.607683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.607659 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:20.612117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.612098 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kff8\" (UniqueName: \"kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8\") pod \"data-science-collector-collector-1\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.700574 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.700551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert\") pod \"57c71310-9786-4924-9db1-09f59ba182dc\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " Oct 31 00:13:20.700708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.700652 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal\" (UniqueName: \"kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal\") pod \"57c71310-9786-4924-9db1-09f59ba182dc\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " Oct 31 00:13:20.700708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.700694 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr\" (UniqueName: \"kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr\") pod \"57c71310-9786-4924-9db1-09f59ba182dc\" (UID: \"57c71310-9786-4924-9db1-09f59ba182dc\") " Oct 31 00:13:20.700913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.700891 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal" pod "57c71310-9786-4924-9db1-09f59ba182dc" (UID: "57c71310-9786-4924-9db1-09f59ba182dc"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:20.701018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.701006 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57c71310-9786-4924-9db1-09f59ba182dc-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.702540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.702520 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert" pod "57c71310-9786-4924-9db1-09f59ba182dc" (UID: "57c71310-9786-4924-9db1-09f59ba182dc"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:20.702691 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.702673 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr" pod "57c71310-9786-4924-9db1-09f59ba182dc" (UID: "57c71310-9786-4924-9db1-09f59ba182dc"). InnerVolumeSpecName "kube-api-access-kbmpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:20.748960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.748935 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:20.801466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.801438 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57c71310-9786-4924-9db1-09f59ba182dc-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:20.801612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:20.801471 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmpr\" (UniqueName: \"kubernetes.io/projected/57c71310-9786-4924-9db1-09f59ba182dc-kube-api-access-kbmpr\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.026059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.025795 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:21.028390 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:21.028363 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4321769a_9e09_41d0_8040_4cfb29dae706.slice/crio-6227b20dfacd153e0b1bd708a7dc47f20286f76387da2b8f26dfcb5fa142b135 WatchSource:0}: Error finding container 6227b20dfacd153e0b1bd708a7dc47f20286f76387da2b8f26dfcb5fa142b135: Status 404 returned error can't find the container with id 6227b20dfacd153e0b1bd708a7dc47f20286f76387da2b8f26dfcb5fa142b135 Oct 31 00:13:21.030552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.030522 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:21.030631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.030575 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:21.319265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.319167 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4321769a-9e09-41d0-8040-4cfb29dae706","Type":"ContainerStarted","Data":"01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714"} Oct 31 00:13:21.319265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.319228 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4321769a-9e09-41d0-8040-4cfb29dae706","Type":"ContainerStarted","Data":"6227b20dfacd153e0b1bd708a7dc47f20286f76387da2b8f26dfcb5fa142b135"} Oct 31 00:13:21.321461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.321436 5073 generic.go:356] "Generic (PLEG): container finished" podID="57c71310-9786-4924-9db1-09f59ba182dc" containerID="8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea" exitCode=0 Oct 31 00:13:21.321579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.321469 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" Oct 31 00:13:21.321579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.321519 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" event={"ID":"57c71310-9786-4924-9db1-09f59ba182dc","Type":"ContainerDied","Data":"8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea"} Oct 31 00:13:21.321579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.321558 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj" event={"ID":"57c71310-9786-4924-9db1-09f59ba182dc","Type":"ContainerDied","Data":"ae780bd680d0b496e79d2b638bde56bae50a92dbe1dc8107421191874fd00b2f"} Oct 31 00:13:21.321579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.321578 5073 scope.go:122] "RemoveContainer" containerID="8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea" Oct 31 00:13:21.330758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.330740 5073 scope.go:122] "RemoveContainer" containerID="8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea" Oct 31 00:13:21.331017 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:21.330999 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea\": container with ID starting with 8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea not found: ID does not exist" containerID="8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea" Oct 31 00:13:21.331070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.331026 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea"} err="failed to get container status \"8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea\": rpc error: code = NotFound desc = could not find container \"8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea\": container with ID starting with 8cd036b958c4ae5abb004f21c565bdba0208c6ef27e90bafddc87cf7d36c0fea not found: ID does not exist" Oct 31 00:13:21.341420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.341383 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.341371451 podStartE2EDuration="1.341371451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:21.339278885 +0000 UTC m=+885.400053584" watchObservedRunningTime="2026-10-31 00:13:21.341371451 +0000 UTC m=+885.402146146" Oct 31 00:13:21.358222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.358201 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj"] Oct 31 00:13:21.366465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:21.366437 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b898d755-wbghj"] Oct 31 00:13:22.058615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:22.058583 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562a3e76-b186-4b7b-9f69-ea57915ec91a" path="/var/lib/kubelet/pods/562a3e76-b186-4b7b-9f69-ea57915ec91a/volumes" Oct 31 00:13:22.059000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:22.058947 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c71310-9786-4924-9db1-09f59ba182dc" path="/var/lib/kubelet/pods/57c71310-9786-4924-9db1-09f59ba182dc/volumes" Oct 31 00:13:30.860035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.859999 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:30.860504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.860213 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4321769a-9e09-41d0-8040-4cfb29dae706" containerName="otc-container" containerID="cri-o://01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714" gracePeriod=30 Oct 31 00:13:30.972076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.972043 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl"] Oct 31 00:13:30.972476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.972459 5073 cpu_manager.go:414] "containerMap: removing container" podUID="57c71310-9786-4924-9db1-09f59ba182dc" containerName="ta-container" Oct 31 00:13:30.972579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.972479 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c71310-9786-4924-9db1-09f59ba182dc" containerName="ta-container" Oct 31 00:13:30.972638 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.972589 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c71310-9786-4924-9db1-09f59ba182dc" containerName="ta-container" Oct 31 00:13:30.975649 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.975629 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:30.984144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.984115 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqvz\" (UniqueName: \"kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:30.984262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.984238 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:30.984329 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.984283 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:30.986774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:30.986747 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl"] Oct 31 00:13:31.084677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.084639 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqvz\" (UniqueName: \"kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.084873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.084691 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.084873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.084833 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.085436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.085418 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.087242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.087219 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.097315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.097287 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqvz\" (UniqueName: \"kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz\") pod \"data-science-collector-targetallocator-6bfd8f89-cwlgl\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.288021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.288000 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:31.561663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.561465 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl"] Oct 31 00:13:31.564372 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:31.564346 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a76cf58_ee5c_47fb_8460_0275fa9c2139.slice/crio-776abd81fa7cc973519a0a02c978f80cf56dd49475a1f366ac21ff2159cfc6b6 WatchSource:0}: Error finding container 776abd81fa7cc973519a0a02c978f80cf56dd49475a1f366ac21ff2159cfc6b6: Status 404 returned error can't find the container with id 776abd81fa7cc973519a0a02c978f80cf56dd49475a1f366ac21ff2159cfc6b6 Oct 31 00:13:31.566589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.566552 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:31.566674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:31.566613 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:32.111926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.111904 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.294754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.294727 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs\") pod \"4321769a-9e09-41d0-8040-4cfb29dae706\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " Oct 31 00:13:32.294935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.294813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal\") pod \"4321769a-9e09-41d0-8040-4cfb29dae706\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " Oct 31 00:13:32.294935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.294849 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs\") pod \"4321769a-9e09-41d0-8040-4cfb29dae706\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " Oct 31 00:13:32.294935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.294898 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert\") pod \"4321769a-9e09-41d0-8040-4cfb29dae706\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " Oct 31 00:13:32.295100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.294947 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8\" (UniqueName: \"kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8\") pod \"4321769a-9e09-41d0-8040-4cfb29dae706\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " Oct 31 00:13:32.295100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.295029 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs\") pod \"4321769a-9e09-41d0-8040-4cfb29dae706\" (UID: \"4321769a-9e09-41d0-8040-4cfb29dae706\") " Oct 31 00:13:32.295302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.295257 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal" pod "4321769a-9e09-41d0-8040-4cfb29dae706" (UID: "4321769a-9e09-41d0-8040-4cfb29dae706"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:32.297356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.297329 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8" pod "4321769a-9e09-41d0-8040-4cfb29dae706" (UID: "4321769a-9e09-41d0-8040-4cfb29dae706"). InnerVolumeSpecName "kube-api-access-4kff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:32.297473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.297423 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs" pod "4321769a-9e09-41d0-8040-4cfb29dae706" (UID: "4321769a-9e09-41d0-8040-4cfb29dae706"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:32.297473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.297448 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert" pod "4321769a-9e09-41d0-8040-4cfb29dae706" (UID: "4321769a-9e09-41d0-8040-4cfb29dae706"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:32.297630 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.297604 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs" pod "4321769a-9e09-41d0-8040-4cfb29dae706" (UID: "4321769a-9e09-41d0-8040-4cfb29dae706"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:32.297877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.297853 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs" pod "4321769a-9e09-41d0-8040-4cfb29dae706" (UID: "4321769a-9e09-41d0-8040-4cfb29dae706"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:32.377400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.377377 5073 generic.go:356] "Generic (PLEG): container finished" podID="4321769a-9e09-41d0-8040-4cfb29dae706" containerID="01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714" exitCode=0 Oct 31 00:13:32.377521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.377433 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.377521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.377463 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4321769a-9e09-41d0-8040-4cfb29dae706","Type":"ContainerDied","Data":"01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714"} Oct 31 00:13:32.377521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.377514 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4321769a-9e09-41d0-8040-4cfb29dae706","Type":"ContainerDied","Data":"6227b20dfacd153e0b1bd708a7dc47f20286f76387da2b8f26dfcb5fa142b135"} Oct 31 00:13:32.377678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.377540 5073 scope.go:122] "RemoveContainer" containerID="01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714" Oct 31 00:13:32.379107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.379089 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" event={"ID":"6a76cf58-ee5c-47fb-8460-0275fa9c2139","Type":"ContainerStarted","Data":"5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213"} Oct 31 00:13:32.379216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.379116 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" event={"ID":"6a76cf58-ee5c-47fb-8460-0275fa9c2139","Type":"ContainerStarted","Data":"776abd81fa7cc973519a0a02c978f80cf56dd49475a1f366ac21ff2159cfc6b6"} Oct 31 00:13:32.379299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.379282 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:32.381227 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.381210 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:32.391275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.389152 5073 scope.go:122] "RemoveContainer" containerID="01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714" Oct 31 00:13:32.391275 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:32.389497 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714\": container with ID starting with 01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714 not found: ID does not exist" containerID="01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714" Oct 31 00:13:32.391275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.389535 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714"} err="failed to get container status \"01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714\": rpc error: code = NotFound desc = could not find container \"01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714\": container with ID starting with 01b5d2c7abd8b2dff1ae212ccce1f456fd9b1ea67ca35f2042716c796c796714 not found: ID does not exist" Oct 31 00:13:32.395910 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.395889 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.395988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.395911 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4321769a-9e09-41d0-8040-4cfb29dae706-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.395988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.395921 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.395988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.395931 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.395988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.395941 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kff8\" (UniqueName: \"kubernetes.io/projected/4321769a-9e09-41d0-8040-4cfb29dae706-kube-api-access-4kff8\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.395988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.395949 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4321769a-9e09-41d0-8040-4cfb29dae706-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.402528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.402484 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" podStartSLOduration=2.402473543 podStartE2EDuration="2.402473543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:32.400961564 +0000 UTC m=+896.461736251" watchObservedRunningTime="2026-10-31 00:13:32.402473543 +0000 UTC m=+896.463248237" Oct 31 00:13:32.418002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.417980 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:32.422599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.422578 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:32.431962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.431943 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw"] Oct 31 00:13:32.432115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.432095 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" podUID="93d53244-bc34-4bb0-8e9c-225e250753c3" containerName="ta-container" containerID="cri-o://d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1" gracePeriod=30 Oct 31 00:13:32.449988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.449964 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:32.450554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.450535 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4321769a-9e09-41d0-8040-4cfb29dae706" containerName="otc-container" Oct 31 00:13:32.450601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.450558 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4321769a-9e09-41d0-8040-4cfb29dae706" containerName="otc-container" Oct 31 00:13:32.450708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.450696 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4321769a-9e09-41d0-8040-4cfb29dae706" containerName="otc-container" Oct 31 00:13:32.453958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.453936 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.458056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.458029 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:32.458056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.458031 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:32.458191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.458080 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:32.458191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.458094 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:32.458647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.458627 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\"" Oct 31 00:13:32.458723 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.458661 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:32.463633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.463615 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:32.598354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.598324 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.598503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.598370 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.598503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.598392 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.598636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.598516 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.598636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.598543 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.598636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.598564 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq28w\" (UniqueName: \"kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.657657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.657634 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:32.699685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.699657 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.699859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.699700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.699859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.699732 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.699859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.699776 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.699859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.699820 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.699859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.699847 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq28w\" (UniqueName: \"kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.700444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.700415 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.703030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.703004 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.703030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.703014 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.703157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.703041 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.703354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.703333 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.712100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.712076 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq28w\" (UniqueName: \"kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w\") pod \"data-science-collector-collector-1\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.800904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.800868 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z\" (UniqueName: \"kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z\") pod \"93d53244-bc34-4bb0-8e9c-225e250753c3\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " Oct 31 00:13:32.801085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.800957 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal\" (UniqueName: \"kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal\") pod \"93d53244-bc34-4bb0-8e9c-225e250753c3\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " Oct 31 00:13:32.801085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.800985 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert\") pod \"93d53244-bc34-4bb0-8e9c-225e250753c3\" (UID: \"93d53244-bc34-4bb0-8e9c-225e250753c3\") " Oct 31 00:13:32.801360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.801330 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal" pod "93d53244-bc34-4bb0-8e9c-225e250753c3" (UID: "93d53244-bc34-4bb0-8e9c-225e250753c3"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:32.803315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.803290 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z" pod "93d53244-bc34-4bb0-8e9c-225e250753c3" (UID: "93d53244-bc34-4bb0-8e9c-225e250753c3"). InnerVolumeSpecName "kube-api-access-87p7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:32.803395 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.803315 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert" pod "93d53244-bc34-4bb0-8e9c-225e250753c3" (UID: "93d53244-bc34-4bb0-8e9c-225e250753c3"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:32.815483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.815466 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:32.902371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.902344 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87p7z\" (UniqueName: \"kubernetes.io/projected/93d53244-bc34-4bb0-8e9c-225e250753c3-kube-api-access-87p7z\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.902371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.902370 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/93d53244-bc34-4bb0-8e9c-225e250753c3-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:32.902541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:32.902382 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/93d53244-bc34-4bb0-8e9c-225e250753c3-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:33.091683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.091649 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:33.094550 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:33.094524 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb937b70_1171_4b44_b8eb_4dde369d8fde.slice/crio-298b6dc196aee0694ad1dc325c3211198ec9272d322127ef1841e37879bb64f0 WatchSource:0}: Error finding container 298b6dc196aee0694ad1dc325c3211198ec9272d322127ef1841e37879bb64f0: Status 404 returned error can't find the container with id 298b6dc196aee0694ad1dc325c3211198ec9272d322127ef1841e37879bb64f0 Oct 31 00:13:33.096825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.096777 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:33.096928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.096863 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:33.385540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.385453 5073 generic.go:356] "Generic (PLEG): container finished" podID="93d53244-bc34-4bb0-8e9c-225e250753c3" containerID="d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1" exitCode=0 Oct 31 00:13:33.385540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.385500 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" Oct 31 00:13:33.386089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.385549 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" event={"ID":"93d53244-bc34-4bb0-8e9c-225e250753c3","Type":"ContainerDied","Data":"d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1"} Oct 31 00:13:33.386089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.385590 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw" event={"ID":"93d53244-bc34-4bb0-8e9c-225e250753c3","Type":"ContainerDied","Data":"4646e07a1535a8f1c8adde5ecda212974b69af187f031820f20c25f95d685417"} Oct 31 00:13:33.386089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.385610 5073 scope.go:122] "RemoveContainer" containerID="d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1" Oct 31 00:13:33.388760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.388733 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db937b70-1171-4b44-b8eb-4dde369d8fde","Type":"ContainerStarted","Data":"8aadb855258c0f13d556ed89be71683d0d9187192b06a157bc66956c30c94c11"} Oct 31 00:13:33.388949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.388804 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db937b70-1171-4b44-b8eb-4dde369d8fde","Type":"ContainerStarted","Data":"298b6dc196aee0694ad1dc325c3211198ec9272d322127ef1841e37879bb64f0"} Oct 31 00:13:33.395734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.395718 5073 scope.go:122] "RemoveContainer" containerID="d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1" Oct 31 00:13:33.396050 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:33.396032 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1\": container with ID starting with d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1 not found: ID does not exist" containerID="d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1" Oct 31 00:13:33.396116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.396059 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1"} err="failed to get container status \"d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1\": rpc error: code = NotFound desc = could not find container \"d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1\": container with ID starting with d968df1889ad6fd7c348876d1e2c5dd2777fd5a4440d5940b914c8e8986a68e1 not found: ID does not exist" Oct 31 00:13:33.409702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.409658 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.409643257 podStartE2EDuration="1.409643257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:33.409257723 +0000 UTC m=+897.470032419" watchObservedRunningTime="2026-10-31 00:13:33.409643257 +0000 UTC m=+897.470417952" Oct 31 00:13:33.425123 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.425044 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw"] Oct 31 00:13:33.436296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:33.436266 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6fd9967d54-d2nfw"] Oct 31 00:13:34.058726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:34.058693 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4321769a-9e09-41d0-8040-4cfb29dae706" path="/var/lib/kubelet/pods/4321769a-9e09-41d0-8040-4cfb29dae706/volumes" Oct 31 00:13:34.059104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:34.059091 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d53244-bc34-4bb0-8e9c-225e250753c3" path="/var/lib/kubelet/pods/93d53244-bc34-4bb0-8e9c-225e250753c3/volumes" Oct 31 00:13:36.135669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:36.135647 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:13:36.139078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:36.139056 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:13:36.149645 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:36.149620 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:13:36.153032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:36.153012 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:13:43.085806 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:43.085747 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:13:43.218868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.218835 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:43.219071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.219050 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="db937b70-1171-4b44-b8eb-4dde369d8fde" containerName="otc-container" containerID="cri-o://8aadb855258c0f13d556ed89be71683d0d9187192b06a157bc66956c30c94c11" gracePeriod=30 Oct 31 00:13:43.317064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.317036 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl"] Oct 31 00:13:43.317383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.317371 5073 cpu_manager.go:414] "containerMap: removing container" podUID="93d53244-bc34-4bb0-8e9c-225e250753c3" containerName="ta-container" Oct 31 00:13:43.317425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.317384 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d53244-bc34-4bb0-8e9c-225e250753c3" containerName="ta-container" Oct 31 00:13:43.317470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.317461 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d53244-bc34-4bb0-8e9c-225e250753c3" containerName="ta-container" Oct 31 00:13:43.322355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.322335 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.333304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.332398 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl"] Oct 31 00:13:43.382285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.382208 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.382285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.382251 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lqd\" (UniqueName: \"kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.382456 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.382437 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.483616 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.483586 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.483762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.483629 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.483762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.483649 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lqd\" (UniqueName: \"kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.484232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.484207 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.486173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.486155 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.496187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.496166 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lqd\" (UniqueName: \"kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd\") pod \"data-science-collector-targetallocator-84696775d4-hzwvl\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.639744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.639669 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:43.914988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.914915 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl"] Oct 31 00:13:43.917962 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:43.917920 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0e19c41_1b49_49bc_8306_49a95bc74c72.slice/crio-8c677785f4f2183e35aabda0904838376a2980f1a461a555d47a2dec90624af6 WatchSource:0}: Error finding container 8c677785f4f2183e35aabda0904838376a2980f1a461a555d47a2dec90624af6: Status 404 returned error can't find the container with id 8c677785f4f2183e35aabda0904838376a2980f1a461a555d47a2dec90624af6 Oct 31 00:13:43.920130 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.920094 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:43.920235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:43.920150 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:44.439834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.439797 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" event={"ID":"b0e19c41-1b49-49bc-8306-49a95bc74c72","Type":"ContainerStarted","Data":"4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96"} Oct 31 00:13:44.439834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.439838 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" event={"ID":"b0e19c41-1b49-49bc-8306-49a95bc74c72","Type":"ContainerStarted","Data":"8c677785f4f2183e35aabda0904838376a2980f1a461a555d47a2dec90624af6"} Oct 31 00:13:44.440313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.439962 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:44.441756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.441736 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:44.459303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.458836 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" podStartSLOduration=1.458822708 podStartE2EDuration="1.458822708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:44.457936467 +0000 UTC m=+908.518711161" watchObservedRunningTime="2026-10-31 00:13:44.458822708 +0000 UTC m=+908.519597404" Oct 31 00:13:44.508557 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.508527 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl"] Oct 31 00:13:44.508746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.508723 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" podUID="6a76cf58-ee5c-47fb-8460-0275fa9c2139" containerName="ta-container" containerID="cri-o://5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213" gracePeriod=30 Oct 31 00:13:44.755594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.755571 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:44.796533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.796507 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal\" (UniqueName: \"kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal\") pod \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " Oct 31 00:13:44.796728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.796551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert\") pod \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " Oct 31 00:13:44.796728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.796657 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz\" (UniqueName: \"kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz\") pod \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\" (UID: \"6a76cf58-ee5c-47fb-8460-0275fa9c2139\") " Oct 31 00:13:44.796998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.796967 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal" pod "6a76cf58-ee5c-47fb-8460-0275fa9c2139" (UID: "6a76cf58-ee5c-47fb-8460-0275fa9c2139"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:44.799207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.799185 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert" pod "6a76cf58-ee5c-47fb-8460-0275fa9c2139" (UID: "6a76cf58-ee5c-47fb-8460-0275fa9c2139"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:44.799300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.799201 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz" pod "6a76cf58-ee5c-47fb-8460-0275fa9c2139" (UID: "6a76cf58-ee5c-47fb-8460-0275fa9c2139"). InnerVolumeSpecName "kube-api-access-6nqvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:44.897628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.897603 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nqvz\" (UniqueName: \"kubernetes.io/projected/6a76cf58-ee5c-47fb-8460-0275fa9c2139-kube-api-access-6nqvz\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:44.897628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.897626 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6a76cf58-ee5c-47fb-8460-0275fa9c2139-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:44.897763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:44.897638 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6a76cf58-ee5c-47fb-8460-0275fa9c2139-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:45.446081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.446044 5073 generic.go:356] "Generic (PLEG): container finished" podID="6a76cf58-ee5c-47fb-8460-0275fa9c2139" containerID="5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213" exitCode=0 Oct 31 00:13:45.446479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.446088 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" Oct 31 00:13:45.446479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.446112 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" event={"ID":"6a76cf58-ee5c-47fb-8460-0275fa9c2139","Type":"ContainerDied","Data":"5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213"} Oct 31 00:13:45.446479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.446148 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl" event={"ID":"6a76cf58-ee5c-47fb-8460-0275fa9c2139","Type":"ContainerDied","Data":"776abd81fa7cc973519a0a02c978f80cf56dd49475a1f366ac21ff2159cfc6b6"} Oct 31 00:13:45.446479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.446174 5073 scope.go:122] "RemoveContainer" containerID="5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213" Oct 31 00:13:45.447984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.447964 5073 generic.go:356] "Generic (PLEG): container finished" podID="db937b70-1171-4b44-b8eb-4dde369d8fde" containerID="8aadb855258c0f13d556ed89be71683d0d9187192b06a157bc66956c30c94c11" exitCode=0 Oct 31 00:13:45.448111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.448048 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db937b70-1171-4b44-b8eb-4dde369d8fde","Type":"ContainerDied","Data":"8aadb855258c0f13d556ed89be71683d0d9187192b06a157bc66956c30c94c11"} Oct 31 00:13:45.455769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.455750 5073 scope.go:122] "RemoveContainer" containerID="5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213" Oct 31 00:13:45.456090 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:45.456070 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213\": container with ID starting with 5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213 not found: ID does not exist" containerID="5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213" Oct 31 00:13:45.456179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.456099 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213"} err="failed to get container status \"5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213\": rpc error: code = NotFound desc = could not find container \"5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213\": container with ID starting with 5be6252ce142b5ea28f80b98de016960866a7c6e67985f597d0de8e0733fc213 not found: ID does not exist" Oct 31 00:13:45.468725 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.468699 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl"] Oct 31 00:13:45.472600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.472581 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bfd8f89-cwlgl"] Oct 31 00:13:45.574222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.574201 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:45.603300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.603272 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs\") pod \"db937b70-1171-4b44-b8eb-4dde369d8fde\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " Oct 31 00:13:45.603419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.603312 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs\") pod \"db937b70-1171-4b44-b8eb-4dde369d8fde\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " Oct 31 00:13:45.603419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.603356 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs\") pod \"db937b70-1171-4b44-b8eb-4dde369d8fde\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " Oct 31 00:13:45.603419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.603400 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w\" (UniqueName: \"kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w\") pod \"db937b70-1171-4b44-b8eb-4dde369d8fde\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " Oct 31 00:13:45.603540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.603497 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert\") pod \"db937b70-1171-4b44-b8eb-4dde369d8fde\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " Oct 31 00:13:45.603540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.603533 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal\" (UniqueName: \"kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal\") pod \"db937b70-1171-4b44-b8eb-4dde369d8fde\" (UID: \"db937b70-1171-4b44-b8eb-4dde369d8fde\") " Oct 31 00:13:45.604360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.604327 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal" pod "db937b70-1171-4b44-b8eb-4dde369d8fde" (UID: "db937b70-1171-4b44-b8eb-4dde369d8fde"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:45.607495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.607158 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs" pod "db937b70-1171-4b44-b8eb-4dde369d8fde" (UID: "db937b70-1171-4b44-b8eb-4dde369d8fde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:45.607639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.607541 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert" pod "db937b70-1171-4b44-b8eb-4dde369d8fde" (UID: "db937b70-1171-4b44-b8eb-4dde369d8fde"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:45.607761 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.607734 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs" pod "db937b70-1171-4b44-b8eb-4dde369d8fde" (UID: "db937b70-1171-4b44-b8eb-4dde369d8fde"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:45.608191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.608160 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs" pod "db937b70-1171-4b44-b8eb-4dde369d8fde" (UID: "db937b70-1171-4b44-b8eb-4dde369d8fde"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:45.608621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.608605 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w" pod "db937b70-1171-4b44-b8eb-4dde369d8fde" (UID: "db937b70-1171-4b44-b8eb-4dde369d8fde"). InnerVolumeSpecName "kube-api-access-xq28w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:45.704468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.704394 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:45.704468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.704420 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db937b70-1171-4b44-b8eb-4dde369d8fde-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:45.704468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.704430 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:45.704468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.704439 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:45.704468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.704448 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db937b70-1171-4b44-b8eb-4dde369d8fde-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:45.704468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:45.704457 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq28w\" (UniqueName: \"kubernetes.io/projected/db937b70-1171-4b44-b8eb-4dde369d8fde-kube-api-access-xq28w\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.058103 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.058070 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a76cf58-ee5c-47fb-8460-0275fa9c2139" path="/var/lib/kubelet/pods/6a76cf58-ee5c-47fb-8460-0275fa9c2139/volumes" Oct 31 00:13:46.454223 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.454141 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.454223 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.454189 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db937b70-1171-4b44-b8eb-4dde369d8fde","Type":"ContainerDied","Data":"298b6dc196aee0694ad1dc325c3211198ec9272d322127ef1841e37879bb64f0"} Oct 31 00:13:46.454742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.454259 5073 scope.go:122] "RemoveContainer" containerID="8aadb855258c0f13d556ed89be71683d0d9187192b06a157bc66956c30c94c11" Oct 31 00:13:46.471981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.471959 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:46.476011 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.475990 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:46.501544 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501519 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:46.501892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501878 5073 cpu_manager.go:414] "containerMap: removing container" podUID="db937b70-1171-4b44-b8eb-4dde369d8fde" containerName="otc-container" Oct 31 00:13:46.501942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501894 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db937b70-1171-4b44-b8eb-4dde369d8fde" containerName="otc-container" Oct 31 00:13:46.501942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501912 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6a76cf58-ee5c-47fb-8460-0275fa9c2139" containerName="ta-container" Oct 31 00:13:46.501942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501917 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a76cf58-ee5c-47fb-8460-0275fa9c2139" containerName="ta-container" Oct 31 00:13:46.502036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501965 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="db937b70-1171-4b44-b8eb-4dde369d8fde" containerName="otc-container" Oct 31 00:13:46.502036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.501975 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a76cf58-ee5c-47fb-8460-0275fa9c2139" containerName="ta-container" Oct 31 00:13:46.506044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.506027 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.510340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.510320 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:46.510464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.510326 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:46.510464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.510342 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:46.510464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.510353 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:46.510657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.510353 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:46.510736 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.510708 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:13:46.515812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.515763 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:46.611029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.611002 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdq5x\" (UniqueName: \"kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.611150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.611074 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.611150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.611130 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.611225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.611168 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.611225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.611201 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.611328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.611306 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712130 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712161 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712181 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdq5x\" (UniqueName: \"kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712408 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712227 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712408 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712243 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712408 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712268 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.712948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.712928 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.715028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.715004 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.715119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.715057 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.715552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.715531 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.715620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.715592 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.724716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.724694 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdq5x\" (UniqueName: \"kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x\") pod \"data-science-collector-collector-1\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:46.819940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:46.819919 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:47.101930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:47.101860 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:47.104580 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:47.104553 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6509a42_7991_4346_ae9c_2f84d5ac88ad.slice/crio-b629238a22ed02dedb688a506a7d80bbbae376c6e22c26b884b701592488d3a9 WatchSource:0}: Error finding container b629238a22ed02dedb688a506a7d80bbbae376c6e22c26b884b701592488d3a9: Status 404 returned error can't find the container with id b629238a22ed02dedb688a506a7d80bbbae376c6e22c26b884b701592488d3a9 Oct 31 00:13:47.106710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:47.106676 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:47.106840 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:47.106744 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:47.467079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:47.466996 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a6509a42-7991-4346-ae9c-2f84d5ac88ad","Type":"ContainerStarted","Data":"7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b"} Oct 31 00:13:47.467079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:47.467069 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a6509a42-7991-4346-ae9c-2f84d5ac88ad","Type":"ContainerStarted","Data":"b629238a22ed02dedb688a506a7d80bbbae376c6e22c26b884b701592488d3a9"} Oct 31 00:13:47.488448 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:47.488390 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.488374534 podStartE2EDuration="1.488374534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:47.487601499 +0000 UTC m=+911.548376194" watchObservedRunningTime="2026-10-31 00:13:47.488374534 +0000 UTC m=+911.549149230" Oct 31 00:13:48.057803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:48.057765 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db937b70-1171-4b44-b8eb-4dde369d8fde" path="/var/lib/kubelet/pods/db937b70-1171-4b44-b8eb-4dde369d8fde/volumes" Oct 31 00:13:55.459703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.459672 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:55.460091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.459885 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a6509a42-7991-4346-ae9c-2f84d5ac88ad" containerName="otc-container" containerID="cri-o://7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b" gracePeriod=30 Oct 31 00:13:55.562064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.562032 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9"] Oct 31 00:13:55.566019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.565995 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.577816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.577753 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9"] Oct 31 00:13:55.682185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.682157 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.682357 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.682205 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.682357 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.682245 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7sd\" (UniqueName: \"kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.782985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.782952 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.783100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.783003 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.783100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.783051 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7sd\" (UniqueName: \"kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.783668 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.783644 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.785868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.785846 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.795131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.795110 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7sd\" (UniqueName: \"kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd\") pod \"data-science-collector-targetallocator-8656589dc9-7hhn9\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:55.879820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:55.879799 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:56.154364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.154253 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9"] Oct 31 00:13:56.157061 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:56.157036 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e905324_1d5c_4879_b6ac_95cd2bdfe853.slice/crio-fbf7471ad6f3407027e71a285ad4ee0488957cb926e94b0d223da1a0d22edef7 WatchSource:0}: Error finding container fbf7471ad6f3407027e71a285ad4ee0488957cb926e94b0d223da1a0d22edef7: Status 404 returned error can't find the container with id fbf7471ad6f3407027e71a285ad4ee0488957cb926e94b0d223da1a0d22edef7 Oct 31 00:13:56.159136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.159107 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:56.159217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.159163 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:56.508145 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.508119 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" event={"ID":"3e905324-1d5c-4879-b6ac-95cd2bdfe853","Type":"ContainerStarted","Data":"0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d"} Oct 31 00:13:56.508549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.508155 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" event={"ID":"3e905324-1d5c-4879-b6ac-95cd2bdfe853","Type":"ContainerStarted","Data":"fbf7471ad6f3407027e71a285ad4ee0488957cb926e94b0d223da1a0d22edef7"} Oct 31 00:13:56.508549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.508310 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:56.509810 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.509794 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:13:56.528022 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.527981 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" podStartSLOduration=1.527969371 podStartE2EDuration="1.527969371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:56.525958697 +0000 UTC m=+920.586733391" watchObservedRunningTime="2026-10-31 00:13:56.527969371 +0000 UTC m=+920.588744056" Oct 31 00:13:56.573436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.573408 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl"] Oct 31 00:13:56.573644 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.573610 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" podUID="b0e19c41-1b49-49bc-8306-49a95bc74c72" containerName="ta-container" containerID="cri-o://4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96" gracePeriod=30 Oct 31 00:13:56.703716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.703694 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:56.792226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.792143 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs\") pod \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " Oct 31 00:13:56.792226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.792198 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert\") pod \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " Oct 31 00:13:56.792445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.792258 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs\") pod \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " Oct 31 00:13:56.792445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.792289 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal\") pod \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " Oct 31 00:13:56.792445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.792316 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x\" (UniqueName: \"kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x\") pod \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " Oct 31 00:13:56.792445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.792355 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs\") pod \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\" (UID: \"a6509a42-7991-4346-ae9c-2f84d5ac88ad\") " Oct 31 00:13:56.793125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.793012 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal" pod "a6509a42-7991-4346-ae9c-2f84d5ac88ad" (UID: "a6509a42-7991-4346-ae9c-2f84d5ac88ad"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:56.795021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.794970 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs" pod "a6509a42-7991-4346-ae9c-2f84d5ac88ad" (UID: "a6509a42-7991-4346-ae9c-2f84d5ac88ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:56.795021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.795013 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs" pod "a6509a42-7991-4346-ae9c-2f84d5ac88ad" (UID: "a6509a42-7991-4346-ae9c-2f84d5ac88ad"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:56.795428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.795406 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs" pod "a6509a42-7991-4346-ae9c-2f84d5ac88ad" (UID: "a6509a42-7991-4346-ae9c-2f84d5ac88ad"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:56.795694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.795671 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert" pod "a6509a42-7991-4346-ae9c-2f84d5ac88ad" (UID: "a6509a42-7991-4346-ae9c-2f84d5ac88ad"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:56.795825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.795698 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x" pod "a6509a42-7991-4346-ae9c-2f84d5ac88ad" (UID: "a6509a42-7991-4346-ae9c-2f84d5ac88ad"). InnerVolumeSpecName "kube-api-access-gdq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:56.840217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.840193 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:56.893141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893118 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert\") pod \"b0e19c41-1b49-49bc-8306-49a95bc74c72\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " Oct 31 00:13:56.893235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893184 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd\" (UniqueName: \"kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd\") pod \"b0e19c41-1b49-49bc-8306-49a95bc74c72\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " Oct 31 00:13:56.893290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893232 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal\") pod \"b0e19c41-1b49-49bc-8306-49a95bc74c72\" (UID: \"b0e19c41-1b49-49bc-8306-49a95bc74c72\") " Oct 31 00:13:56.893412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893391 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.893412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893413 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.893575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893428 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdq5x\" (UniqueName: \"kubernetes.io/projected/a6509a42-7991-4346-ae9c-2f84d5ac88ad-kube-api-access-gdq5x\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.893575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893440 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.893575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893452 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.893575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893466 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a6509a42-7991-4346-ae9c-2f84d5ac88ad-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.893706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.893598 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal" pod "b0e19c41-1b49-49bc-8306-49a95bc74c72" (UID: "b0e19c41-1b49-49bc-8306-49a95bc74c72"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:56.895285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.895262 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd" pod "b0e19c41-1b49-49bc-8306-49a95bc74c72" (UID: "b0e19c41-1b49-49bc-8306-49a95bc74c72"). InnerVolumeSpecName "kube-api-access-f4lqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:56.895364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.895332 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert" pod "b0e19c41-1b49-49bc-8306-49a95bc74c72" (UID: "b0e19c41-1b49-49bc-8306-49a95bc74c72"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:56.994803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.994764 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b0e19c41-1b49-49bc-8306-49a95bc74c72-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.994894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.994807 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lqd\" (UniqueName: \"kubernetes.io/projected/b0e19c41-1b49-49bc-8306-49a95bc74c72-kube-api-access-f4lqd\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:56.994894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:56.994819 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b0e19c41-1b49-49bc-8306-49a95bc74c72-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:57.514141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.514111 5073 generic.go:356] "Generic (PLEG): container finished" podID="b0e19c41-1b49-49bc-8306-49a95bc74c72" containerID="4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96" exitCode=0 Oct 31 00:13:57.514611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.514165 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" Oct 31 00:13:57.514611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.514203 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" event={"ID":"b0e19c41-1b49-49bc-8306-49a95bc74c72","Type":"ContainerDied","Data":"4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96"} Oct 31 00:13:57.514611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.514254 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl" event={"ID":"b0e19c41-1b49-49bc-8306-49a95bc74c72","Type":"ContainerDied","Data":"8c677785f4f2183e35aabda0904838376a2980f1a461a555d47a2dec90624af6"} Oct 31 00:13:57.514611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.514273 5073 scope.go:122] "RemoveContainer" containerID="4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96" Oct 31 00:13:57.515659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.515635 5073 generic.go:356] "Generic (PLEG): container finished" podID="a6509a42-7991-4346-ae9c-2f84d5ac88ad" containerID="7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b" exitCode=0 Oct 31 00:13:57.515751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.515671 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.515751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.515724 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a6509a42-7991-4346-ae9c-2f84d5ac88ad","Type":"ContainerDied","Data":"7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b"} Oct 31 00:13:57.515888 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.515767 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a6509a42-7991-4346-ae9c-2f84d5ac88ad","Type":"ContainerDied","Data":"b629238a22ed02dedb688a506a7d80bbbae376c6e22c26b884b701592488d3a9"} Oct 31 00:13:57.524992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.524975 5073 scope.go:122] "RemoveContainer" containerID="4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96" Oct 31 00:13:57.525351 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:57.525319 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96\": container with ID starting with 4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96 not found: ID does not exist" containerID="4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96" Oct 31 00:13:57.525444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.525356 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96"} err="failed to get container status \"4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96\": rpc error: code = NotFound desc = could not find container \"4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96\": container with ID starting with 4e3ce36669cba1e62735b85b07469d1257b3bdf9c6661d6725b4b91281488a96 not found: ID does not exist" Oct 31 00:13:57.525444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.525371 5073 scope.go:122] "RemoveContainer" containerID="7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b" Oct 31 00:13:57.535965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.535009 5073 scope.go:122] "RemoveContainer" containerID="7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b" Oct 31 00:13:57.536135 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:13:57.536117 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b\": container with ID starting with 7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b not found: ID does not exist" containerID="7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b" Oct 31 00:13:57.536176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.536142 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b"} err="failed to get container status \"7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b\": rpc error: code = NotFound desc = could not find container \"7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b\": container with ID starting with 7d55c889ae112f2ebfe8efcd373feba4b4afe136e30f2bc514c5a3a19283911b not found: ID does not exist" Oct 31 00:13:57.539165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.539146 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:57.544126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.544097 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:57.558678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.558655 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl"] Oct 31 00:13:57.568200 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.568175 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-84696775d4-hzwvl"] Oct 31 00:13:57.572446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572429 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:57.572807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572775 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b0e19c41-1b49-49bc-8306-49a95bc74c72" containerName="ta-container" Oct 31 00:13:57.572862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572809 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e19c41-1b49-49bc-8306-49a95bc74c72" containerName="ta-container" Oct 31 00:13:57.572862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572818 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a6509a42-7991-4346-ae9c-2f84d5ac88ad" containerName="otc-container" Oct 31 00:13:57.572862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572824 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6509a42-7991-4346-ae9c-2f84d5ac88ad" containerName="otc-container" Oct 31 00:13:57.572964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572880 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e19c41-1b49-49bc-8306-49a95bc74c72" containerName="ta-container" Oct 31 00:13:57.572964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.572886 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6509a42-7991-4346-ae9c-2f84d5ac88ad" containerName="otc-container" Oct 31 00:13:57.576900 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.576885 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.581055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.581037 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:57.581135 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.581113 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:57.581191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.581160 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:57.581484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.581469 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:57.581543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.581483 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:13:57.581543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.581499 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:57.588032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.588012 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:57.701243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.701219 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.701378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.701249 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.701378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.701275 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.701378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.701335 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.701508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.701398 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.701508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.701444 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndsc\" (UniqueName: \"kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.802507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.802441 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.802507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.802494 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.802669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.802536 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.802669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.802566 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.802669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.802617 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndsc\" (UniqueName: \"kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.802915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.802811 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.803657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.803631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.805302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.805277 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.805405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.805374 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.805575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.805551 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.805637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.805574 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.815164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.815142 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndsc\" (UniqueName: \"kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc\") pod \"data-science-collector-collector-1\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:57.890018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:57.889997 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:13:58.059085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.059020 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6509a42-7991-4346-ae9c-2f84d5ac88ad" path="/var/lib/kubelet/pods/a6509a42-7991-4346-ae9c-2f84d5ac88ad/volumes" Oct 31 00:13:58.059405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.059392 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e19c41-1b49-49bc-8306-49a95bc74c72" path="/var/lib/kubelet/pods/b0e19c41-1b49-49bc-8306-49a95bc74c72/volumes" Oct 31 00:13:58.165443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.165419 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:13:58.167968 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:13:58.167943 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd386c907_d80f_4d17_bfbe_638f4d2292a6.slice/crio-81754fb9a0bb1471a2fd37aadd1534cac934131f70e4de239686b1d02704cda6 WatchSource:0}: Error finding container 81754fb9a0bb1471a2fd37aadd1534cac934131f70e4de239686b1d02704cda6: Status 404 returned error can't find the container with id 81754fb9a0bb1471a2fd37aadd1534cac934131f70e4de239686b1d02704cda6 Oct 31 00:13:58.170043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.170009 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:58.170099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.170070 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:13:58.523675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.523637 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d386c907-d80f-4d17-bfbe-638f4d2292a6","Type":"ContainerStarted","Data":"f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54"} Oct 31 00:13:58.524168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.523701 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d386c907-d80f-4d17-bfbe-638f4d2292a6","Type":"ContainerStarted","Data":"81754fb9a0bb1471a2fd37aadd1534cac934131f70e4de239686b1d02704cda6"} Oct 31 00:13:58.544627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:13:58.544584 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.5445693600000001 podStartE2EDuration="1.54456936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:58.543512392 +0000 UTC m=+922.604287087" watchObservedRunningTime="2026-10-31 00:13:58.54456936 +0000 UTC m=+922.605344054" Oct 31 00:14:07.656128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.656099 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:07.656540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.656261 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d386c907-d80f-4d17-bfbe-638f4d2292a6" containerName="otc-container" containerID="cri-o://f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54" gracePeriod=30 Oct 31 00:14:07.770556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.770521 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h"] Oct 31 00:14:07.774209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.774188 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.783989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.783964 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h"] Oct 31 00:14:07.884120 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.884098 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.884242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.884145 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.884286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.884250 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.984934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.984908 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.985019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.984952 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.985019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.984989 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.985523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.985504 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.987589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.987566 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:07.996978 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:07.996955 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl\") pod \"data-science-collector-targetallocator-7f957fdbc4-q4r6h\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:08.086654 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.086629 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:08.361483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.361382 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h"] Oct 31 00:14:08.363577 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:08.363552 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeec5c2f_7473_4113_ae3e_1cacefe7ee3b.slice/crio-0cee8e06bc583471799fad15df4a9b6de756a42adc72fd4ed69a9266508cb552 WatchSource:0}: Error finding container 0cee8e06bc583471799fad15df4a9b6de756a42adc72fd4ed69a9266508cb552: Status 404 returned error can't find the container with id 0cee8e06bc583471799fad15df4a9b6de756a42adc72fd4ed69a9266508cb552 Oct 31 00:14:08.365664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.365633 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:08.365731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.365692 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:08.574847 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.574817 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" event={"ID":"feec5c2f-7473-4113-ae3e-1cacefe7ee3b","Type":"ContainerStarted","Data":"d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e"} Oct 31 00:14:08.575005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.574855 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" event={"ID":"feec5c2f-7473-4113-ae3e-1cacefe7ee3b","Type":"ContainerStarted","Data":"0cee8e06bc583471799fad15df4a9b6de756a42adc72fd4ed69a9266508cb552"} Oct 31 00:14:08.575005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.574975 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:08.576777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.576750 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:08.595692 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.595640 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" podStartSLOduration=1.59556337 podStartE2EDuration="1.59556337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:08.594572074 +0000 UTC m=+932.655346770" watchObservedRunningTime="2026-10-31 00:14:08.59556337 +0000 UTC m=+932.656338066" Oct 31 00:14:08.646634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.646568 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9"] Oct 31 00:14:08.646755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.646720 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" podUID="3e905324-1d5c-4879-b6ac-95cd2bdfe853" containerName="ta-container" containerID="cri-o://0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d" gracePeriod=30 Oct 31 00:14:08.886732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.886709 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:14:08.937769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.937746 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:08.994592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994430 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal\") pod \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " Oct 31 00:14:08.994592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994499 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc\" (UniqueName: \"kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc\") pod \"d386c907-d80f-4d17-bfbe-638f4d2292a6\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " Oct 31 00:14:08.994592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994538 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs\") pod \"d386c907-d80f-4d17-bfbe-638f4d2292a6\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994601 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd\" (UniqueName: \"kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd\") pod \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994634 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs\") pod \"d386c907-d80f-4d17-bfbe-638f4d2292a6\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994692 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert\") pod \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\" (UID: \"3e905324-1d5c-4879-b6ac-95cd2bdfe853\") " Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994726 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert\") pod \"d386c907-d80f-4d17-bfbe-638f4d2292a6\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994771 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal\") pod \"d386c907-d80f-4d17-bfbe-638f4d2292a6\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994798 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal" pod "3e905324-1d5c-4879-b6ac-95cd2bdfe853" (UID: "3e905324-1d5c-4879-b6ac-95cd2bdfe853"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:08.994871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.994833 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs\") pod \"d386c907-d80f-4d17-bfbe-638f4d2292a6\" (UID: \"d386c907-d80f-4d17-bfbe-638f4d2292a6\") " Oct 31 00:14:08.995284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.995118 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e905324-1d5c-4879-b6ac-95cd2bdfe853-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:08.995640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.995553 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal" pod "d386c907-d80f-4d17-bfbe-638f4d2292a6" (UID: "d386c907-d80f-4d17-bfbe-638f4d2292a6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:08.997667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.997629 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs" pod "d386c907-d80f-4d17-bfbe-638f4d2292a6" (UID: "d386c907-d80f-4d17-bfbe-638f4d2292a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:08.998282 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.998162 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs" pod "d386c907-d80f-4d17-bfbe-638f4d2292a6" (UID: "d386c907-d80f-4d17-bfbe-638f4d2292a6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:08.998282 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.998249 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs" pod "d386c907-d80f-4d17-bfbe-638f4d2292a6" (UID: "d386c907-d80f-4d17-bfbe-638f4d2292a6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:08.998441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.998299 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd" pod "3e905324-1d5c-4879-b6ac-95cd2bdfe853" (UID: "3e905324-1d5c-4879-b6ac-95cd2bdfe853"). InnerVolumeSpecName "kube-api-access-xm7sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:08.998583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.998555 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert" pod "3e905324-1d5c-4879-b6ac-95cd2bdfe853" (UID: "3e905324-1d5c-4879-b6ac-95cd2bdfe853"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:08.998653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.998624 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert" pod "d386c907-d80f-4d17-bfbe-638f4d2292a6" (UID: "d386c907-d80f-4d17-bfbe-638f4d2292a6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:08.999221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:08.999200 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc" pod "d386c907-d80f-4d17-bfbe-638f4d2292a6" (UID: "d386c907-d80f-4d17-bfbe-638f4d2292a6"). InnerVolumeSpecName "kube-api-access-nndsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:09.096413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096388 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndsc\" (UniqueName: \"kubernetes.io/projected/d386c907-d80f-4d17-bfbe-638f4d2292a6-kube-api-access-nndsc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096411 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096421 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm7sd\" (UniqueName: \"kubernetes.io/projected/3e905324-1d5c-4879-b6ac-95cd2bdfe853-kube-api-access-xm7sd\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096430 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096441 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e905324-1d5c-4879-b6ac-95cd2bdfe853-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096449 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096458 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d386c907-d80f-4d17-bfbe-638f4d2292a6-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.096519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.096466 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d386c907-d80f-4d17-bfbe-638f4d2292a6-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:09.580517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.580485 5073 generic.go:356] "Generic (PLEG): container finished" podID="3e905324-1d5c-4879-b6ac-95cd2bdfe853" containerID="0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d" exitCode=0 Oct 31 00:14:09.580705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.580542 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" Oct 31 00:14:09.580705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.580576 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" event={"ID":"3e905324-1d5c-4879-b6ac-95cd2bdfe853","Type":"ContainerDied","Data":"0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d"} Oct 31 00:14:09.580705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.580623 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9" event={"ID":"3e905324-1d5c-4879-b6ac-95cd2bdfe853","Type":"ContainerDied","Data":"fbf7471ad6f3407027e71a285ad4ee0488957cb926e94b0d223da1a0d22edef7"} Oct 31 00:14:09.580705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.580645 5073 scope.go:122] "RemoveContainer" containerID="0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d" Oct 31 00:14:09.581920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.581899 5073 generic.go:356] "Generic (PLEG): container finished" podID="d386c907-d80f-4d17-bfbe-638f4d2292a6" containerID="f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54" exitCode=0 Oct 31 00:14:09.582001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.581964 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.582043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.581987 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d386c907-d80f-4d17-bfbe-638f4d2292a6","Type":"ContainerDied","Data":"f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54"} Oct 31 00:14:09.582076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.582062 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d386c907-d80f-4d17-bfbe-638f4d2292a6","Type":"ContainerDied","Data":"81754fb9a0bb1471a2fd37aadd1534cac934131f70e4de239686b1d02704cda6"} Oct 31 00:14:09.589346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.589328 5073 scope.go:122] "RemoveContainer" containerID="0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d" Oct 31 00:14:09.589622 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:09.589600 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d\": container with ID starting with 0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d not found: ID does not exist" containerID="0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d" Oct 31 00:14:09.589727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.589628 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d"} err="failed to get container status \"0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d\": rpc error: code = NotFound desc = could not find container \"0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d\": container with ID starting with 0ae9a2379bedd2ce165524baf325286c43b5fc8925f3e7d906aac70f2345be2d not found: ID does not exist" Oct 31 00:14:09.589727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.589644 5073 scope.go:122] "RemoveContainer" containerID="f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54" Oct 31 00:14:09.597551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.597534 5073 scope.go:122] "RemoveContainer" containerID="f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54" Oct 31 00:14:09.597800 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:09.597762 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54\": container with ID starting with f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54 not found: ID does not exist" containerID="f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54" Oct 31 00:14:09.597864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.597802 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54"} err="failed to get container status \"f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54\": rpc error: code = NotFound desc = could not find container \"f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54\": container with ID starting with f1f45b07dc2e3da949e30b98d7cddc8493ad085d901ccdff784f4433bdd97a54 not found: ID does not exist" Oct 31 00:14:09.606331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.606309 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:09.612513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.612494 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:09.623396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.623376 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9"] Oct 31 00:14:09.626090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.626067 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-8656589dc9-7hhn9"] Oct 31 00:14:09.639235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639214 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:09.639535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639524 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d386c907-d80f-4d17-bfbe-638f4d2292a6" containerName="otc-container" Oct 31 00:14:09.639577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639537 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d386c907-d80f-4d17-bfbe-638f4d2292a6" containerName="otc-container" Oct 31 00:14:09.639577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639555 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3e905324-1d5c-4879-b6ac-95cd2bdfe853" containerName="ta-container" Oct 31 00:14:09.639577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639560 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e905324-1d5c-4879-b6ac-95cd2bdfe853" containerName="ta-container" Oct 31 00:14:09.639676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639636 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d386c907-d80f-4d17-bfbe-638f4d2292a6" containerName="otc-container" Oct 31 00:14:09.639676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.639644 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e905324-1d5c-4879-b6ac-95cd2bdfe853" containerName="ta-container" Oct 31 00:14:09.643821 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.643806 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.647655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.647623 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:09.647655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.647625 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:09.650244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.650168 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:09.654150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.654129 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:09.655598 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.655580 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:09.700959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.700930 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.701070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.700966 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.701070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.701007 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.701167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.701146 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.701204 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.701185 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lvw\" (UniqueName: \"kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.701234 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.701208 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.747807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.747774 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:09.747807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.747795 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:09.802253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.802224 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.802361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.802258 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lvw\" (UniqueName: \"kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.802361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.802307 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.802361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.802349 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.802524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.802371 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.802524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.802399 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.803383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.803355 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.804989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.804962 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.805087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.805066 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.805135 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.805121 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.805216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.805196 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.814799 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.814763 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lvw\" (UniqueName: \"kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw\") pod \"data-science-collector-collector-1\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:09.956587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:09.956535 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:10.059110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.059074 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e905324-1d5c-4879-b6ac-95cd2bdfe853" path="/var/lib/kubelet/pods/3e905324-1d5c-4879-b6ac-95cd2bdfe853/volumes" Oct 31 00:14:10.059518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.059498 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d386c907-d80f-4d17-bfbe-638f4d2292a6" path="/var/lib/kubelet/pods/d386c907-d80f-4d17-bfbe-638f4d2292a6/volumes" Oct 31 00:14:10.235467 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.235434 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:10.241551 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:10.241525 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6220bd_52ae_48a5_8995_1f7c2ce8200d.slice/crio-c5ee8bcd705729821496911025758d5509faba8a915f2766f3a6fbd4c31e2317 WatchSource:0}: Error finding container c5ee8bcd705729821496911025758d5509faba8a915f2766f3a6fbd4c31e2317: Status 404 returned error can't find the container with id c5ee8bcd705729821496911025758d5509faba8a915f2766f3a6fbd4c31e2317 Oct 31 00:14:10.243728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.243692 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:10.243830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.243750 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:10.590316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.590223 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fe6220bd-52ae-48a5-8995-1f7c2ce8200d","Type":"ContainerStarted","Data":"a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0"} Oct 31 00:14:10.590316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.590282 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fe6220bd-52ae-48a5-8995-1f7c2ce8200d","Type":"ContainerStarted","Data":"c5ee8bcd705729821496911025758d5509faba8a915f2766f3a6fbd4c31e2317"} Oct 31 00:14:10.611378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:10.611336 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.611322501 podStartE2EDuration="1.611322501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:10.609499873 +0000 UTC m=+934.670274570" watchObservedRunningTime="2026-10-31 00:14:10.611322501 +0000 UTC m=+934.672097196" Oct 31 00:14:13.085239 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:13.085201 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:14:19.951020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:19.950933 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g"] Oct 31 00:14:19.954552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:19.954530 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:19.963152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:19.963126 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g"] Oct 31 00:14:20.086441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.086410 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.086624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.086476 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.086624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.086556 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtm6j\" (UniqueName: \"kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.187255 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.187224 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.187255 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.187256 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtm6j\" (UniqueName: \"kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.187474 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.187296 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.187924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.187901 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.189857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.189839 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.199586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.199562 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtm6j\" (UniqueName: \"kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j\") pod \"data-science-collector-targetallocator-567d597c85-gw89g\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.268594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.268576 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.545145 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.545071 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g"] Oct 31 00:14:20.548277 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:20.548246 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda7906d2_b772_4472_b5fd_09cdd5512a4c.slice/crio-8a964dbe5e680f9302e24c1067fc8f21962f60a1f8b2edc90f89fcc9c6229b3f WatchSource:0}: Error finding container 8a964dbe5e680f9302e24c1067fc8f21962f60a1f8b2edc90f89fcc9c6229b3f: Status 404 returned error can't find the container with id 8a964dbe5e680f9302e24c1067fc8f21962f60a1f8b2edc90f89fcc9c6229b3f Oct 31 00:14:20.550283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.550246 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:20.550377 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.550308 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:20.651394 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.651366 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" event={"ID":"da7906d2-b772-4472-b5fd-09cdd5512a4c","Type":"ContainerStarted","Data":"e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92"} Oct 31 00:14:20.651561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.651404 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" event={"ID":"da7906d2-b772-4472-b5fd-09cdd5512a4c","Type":"ContainerStarted","Data":"8a964dbe5e680f9302e24c1067fc8f21962f60a1f8b2edc90f89fcc9c6229b3f"} Oct 31 00:14:20.651561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.651534 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:20.652422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.652392 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" podUID="da7906d2-b772-4472-b5fd-09cdd5512a4c" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.83:8080/readyz\": dial tcp 10.132.0.83:8080: connect: connection refused" Oct 31 00:14:20.670672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:20.670628 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" podStartSLOduration=1.670616999 podStartE2EDuration="1.670616999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:20.670166699 +0000 UTC m=+944.730941395" watchObservedRunningTime="2026-10-31 00:14:20.670616999 +0000 UTC m=+944.731391685" Oct 31 00:14:21.657563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:21.657536 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:21.706370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:21.706340 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h"] Oct 31 00:14:21.706549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:21.706521 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" podUID="feec5c2f-7473-4113-ae3e-1cacefe7ee3b" containerName="ta-container" containerID="cri-o://d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e" gracePeriod=30 Oct 31 00:14:21.958626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:21.958602 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:22.104547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.104509 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert\") pod \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " Oct 31 00:14:22.104715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.104595 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal\" (UniqueName: \"kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal\") pod \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " Oct 31 00:14:22.104715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.104627 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl\") pod \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\" (UID: \"feec5c2f-7473-4113-ae3e-1cacefe7ee3b\") " Oct 31 00:14:22.104962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.104939 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal" pod "feec5c2f-7473-4113-ae3e-1cacefe7ee3b" (UID: "feec5c2f-7473-4113-ae3e-1cacefe7ee3b"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:22.106868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.106830 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert" pod "feec5c2f-7473-4113-ae3e-1cacefe7ee3b" (UID: "feec5c2f-7473-4113-ae3e-1cacefe7ee3b"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:22.106868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.106858 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl" pod "feec5c2f-7473-4113-ae3e-1cacefe7ee3b" (UID: "feec5c2f-7473-4113-ae3e-1cacefe7ee3b"). InnerVolumeSpecName "kube-api-access-mtcrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:22.206174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.206115 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:22.206174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.206141 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-kube-api-access-mtcrl\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:22.206174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.206151 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/feec5c2f-7473-4113-ae3e-1cacefe7ee3b-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:22.662662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.662634 5073 generic.go:356] "Generic (PLEG): container finished" podID="feec5c2f-7473-4113-ae3e-1cacefe7ee3b" containerID="d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e" exitCode=0 Oct 31 00:14:22.663110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.662678 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" Oct 31 00:14:22.663110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.662718 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" event={"ID":"feec5c2f-7473-4113-ae3e-1cacefe7ee3b","Type":"ContainerDied","Data":"d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e"} Oct 31 00:14:22.663110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.662756 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h" event={"ID":"feec5c2f-7473-4113-ae3e-1cacefe7ee3b","Type":"ContainerDied","Data":"0cee8e06bc583471799fad15df4a9b6de756a42adc72fd4ed69a9266508cb552"} Oct 31 00:14:22.663110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.662772 5073 scope.go:122] "RemoveContainer" containerID="d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e" Oct 31 00:14:22.671837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.671821 5073 scope.go:122] "RemoveContainer" containerID="d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e" Oct 31 00:14:22.672107 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:22.672085 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e\": container with ID starting with d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e not found: ID does not exist" containerID="d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e" Oct 31 00:14:22.672176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.672114 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e"} err="failed to get container status \"d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e\": rpc error: code = NotFound desc = could not find container \"d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e\": container with ID starting with d1a18541231472826cb965538159905e593a076d4c1c957a7d2c4d5670a40a0e not found: ID does not exist" Oct 31 00:14:22.683884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.683866 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h"] Oct 31 00:14:22.688209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:22.688190 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7f957fdbc4-q4r6h"] Oct 31 00:14:24.058702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:24.058672 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feec5c2f-7473-4113-ae3e-1cacefe7ee3b" path="/var/lib/kubelet/pods/feec5c2f-7473-4113-ae3e-1cacefe7ee3b/volumes" Oct 31 00:14:26.001466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:26.001430 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:26.002039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:26.001708 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="fe6220bd-52ae-48a5-8995-1f7c2ce8200d" containerName="otc-container" containerID="cri-o://a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0" gracePeriod=30 Oct 31 00:14:28.354827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.354802 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.459907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.459829 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal\") pod \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " Oct 31 00:14:28.459907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.459886 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert\") pod \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " Oct 31 00:14:28.460127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.459911 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs\") pod \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " Oct 31 00:14:28.460127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.459978 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw\" (UniqueName: \"kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw\") pod \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " Oct 31 00:14:28.460127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.460007 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs\") pod \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " Oct 31 00:14:28.460127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.460023 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs\") pod \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\" (UID: \"fe6220bd-52ae-48a5-8995-1f7c2ce8200d\") " Oct 31 00:14:28.460339 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.460295 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal" pod "fe6220bd-52ae-48a5-8995-1f7c2ce8200d" (UID: "fe6220bd-52ae-48a5-8995-1f7c2ce8200d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:28.462385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.462350 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs" pod "fe6220bd-52ae-48a5-8995-1f7c2ce8200d" (UID: "fe6220bd-52ae-48a5-8995-1f7c2ce8200d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:28.462521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.462399 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs" pod "fe6220bd-52ae-48a5-8995-1f7c2ce8200d" (UID: "fe6220bd-52ae-48a5-8995-1f7c2ce8200d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:28.462521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.462513 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw" pod "fe6220bd-52ae-48a5-8995-1f7c2ce8200d" (UID: "fe6220bd-52ae-48a5-8995-1f7c2ce8200d"). InnerVolumeSpecName "kube-api-access-s9lvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:28.462830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.462803 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert" pod "fe6220bd-52ae-48a5-8995-1f7c2ce8200d" (UID: "fe6220bd-52ae-48a5-8995-1f7c2ce8200d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:28.462929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.462871 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs" pod "fe6220bd-52ae-48a5-8995-1f7c2ce8200d" (UID: "fe6220bd-52ae-48a5-8995-1f7c2ce8200d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:28.560606 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.560579 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lvw\" (UniqueName: \"kubernetes.io/projected/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-kube-api-access-s9lvw\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:28.560606 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.560603 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:28.560749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.560614 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:28.560749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.560624 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:28.560749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.560635 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:28.560749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.560644 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6220bd-52ae-48a5-8995-1f7c2ce8200d-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:28.693811 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.693770 5073 generic.go:356] "Generic (PLEG): container finished" podID="fe6220bd-52ae-48a5-8995-1f7c2ce8200d" containerID="a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0" exitCode=0 Oct 31 00:14:28.693933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.693839 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.693933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.693864 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fe6220bd-52ae-48a5-8995-1f7c2ce8200d","Type":"ContainerDied","Data":"a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0"} Oct 31 00:14:28.693933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.693921 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fe6220bd-52ae-48a5-8995-1f7c2ce8200d","Type":"ContainerDied","Data":"c5ee8bcd705729821496911025758d5509faba8a915f2766f3a6fbd4c31e2317"} Oct 31 00:14:28.694024 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.693945 5073 scope.go:122] "RemoveContainer" containerID="a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0" Oct 31 00:14:28.703541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.703525 5073 scope.go:122] "RemoveContainer" containerID="a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0" Oct 31 00:14:28.703833 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:28.703814 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0\": container with ID starting with a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0 not found: ID does not exist" containerID="a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0" Oct 31 00:14:28.703916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.703841 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0"} err="failed to get container status \"a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0\": rpc error: code = NotFound desc = could not find container \"a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0\": container with ID starting with a75ff2d75928e603b7c318c433dccbe1d8dceb1040fd14693e81d6f55a6c14a0 not found: ID does not exist" Oct 31 00:14:28.716088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.716035 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:28.721029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.721009 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:28.746462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746442 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:28.746775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746763 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fe6220bd-52ae-48a5-8995-1f7c2ce8200d" containerName="otc-container" Oct 31 00:14:28.746848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746776 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6220bd-52ae-48a5-8995-1f7c2ce8200d" containerName="otc-container" Oct 31 00:14:28.746848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746816 5073 cpu_manager.go:414] "containerMap: removing container" podUID="feec5c2f-7473-4113-ae3e-1cacefe7ee3b" containerName="ta-container" Oct 31 00:14:28.746848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746821 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feec5c2f-7473-4113-ae3e-1cacefe7ee3b" containerName="ta-container" Oct 31 00:14:28.746964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746890 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6220bd-52ae-48a5-8995-1f7c2ce8200d" containerName="otc-container" Oct 31 00:14:28.746964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.746900 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="feec5c2f-7473-4113-ae3e-1cacefe7ee3b" containerName="ta-container" Oct 31 00:14:28.751115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.751100 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.755311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.755280 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:28.755396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.755364 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:28.755778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.755760 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:28.755778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.755768 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:28.755778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.755768 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:28.755998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.755772 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:28.763232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.763171 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:28.864256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.864232 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.864387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.864266 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.864387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.864344 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.864387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.864378 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrfxq\" (UniqueName: \"kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.864506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.864402 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.864506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.864458 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.964882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.964857 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.965006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.964888 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.965167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.965148 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.965220 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.965203 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.965293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.965279 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.965356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.965312 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrfxq\" (UniqueName: \"kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.966238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.966167 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.967718 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.967694 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.967822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.967719 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.967822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.967735 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.967822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.967719 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:28.977839 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:28.977822 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrfxq\" (UniqueName: \"kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq\") pod \"data-science-collector-collector-1\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:29.063642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.063621 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:29.343747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.343655 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:29.346202 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:29.346172 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff51977_fa8e_4050_a2cc_9dfbcb2f63c8.slice/crio-d9112990c429751c92bed1cd78f15818e1236738bc679d5ef7e5c7a082b12eb2 WatchSource:0}: Error finding container d9112990c429751c92bed1cd78f15818e1236738bc679d5ef7e5c7a082b12eb2: Status 404 returned error can't find the container with id d9112990c429751c92bed1cd78f15818e1236738bc679d5ef7e5c7a082b12eb2 Oct 31 00:14:29.348379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.348348 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:29.348455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.348404 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:29.700563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.700477 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8","Type":"ContainerStarted","Data":"d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce"} Oct 31 00:14:29.700994 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.700576 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8","Type":"ContainerStarted","Data":"d9112990c429751c92bed1cd78f15818e1236738bc679d5ef7e5c7a082b12eb2"} Oct 31 00:14:29.722112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:29.722058 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7220453930000001 podStartE2EDuration="1.722045393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:29.719767402 +0000 UTC m=+953.780542100" watchObservedRunningTime="2026-10-31 00:14:29.722045393 +0000 UTC m=+953.782820088" Oct 31 00:14:30.058458 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:30.058427 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6220bd-52ae-48a5-8995-1f7c2ce8200d" path="/var/lib/kubelet/pods/fe6220bd-52ae-48a5-8995-1f7c2ce8200d/volumes" Oct 31 00:14:31.932260 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:31.932056 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:31.933057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:31.932993 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" containerName="otc-container" containerID="cri-o://d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce" gracePeriod=30 Oct 31 00:14:32.021533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.021503 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh"] Oct 31 00:14:32.025000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.024981 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.036866 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.036839 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh"] Oct 31 00:14:32.175944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.175924 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.191641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.191581 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.191743 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.191696 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2m9d\" (UniqueName: \"kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.191800 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.191740 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.292576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292538 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs\") pod \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " Oct 31 00:14:32.292576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292573 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq\" (UniqueName: \"kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq\") pod \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " Oct 31 00:14:32.292762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292616 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal\") pod \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " Oct 31 00:14:32.292762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292666 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs\") pod \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " Oct 31 00:14:32.292762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292711 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert\") pod \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " Oct 31 00:14:32.292762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292754 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs\") pod \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\" (UID: \"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8\") " Oct 31 00:14:32.292980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.292943 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.293041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.293013 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.293828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.293265 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2m9d\" (UniqueName: \"kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.293828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.293421 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal" pod "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" (UID: "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:32.294141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.294100 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.295186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.295165 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert" pod "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" (UID: "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:32.295506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.295479 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs" pod "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" (UID: "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:32.295603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.295580 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq" pod "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" (UID: "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8"). InnerVolumeSpecName "kube-api-access-rrfxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:32.295642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.295632 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs" pod "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" (UID: "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:32.295767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.295747 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.296096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.296074 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs" pod "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" (UID: "aff51977-fa8e-4050-a2cc-9dfbcb2f63c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:32.305568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.305548 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2m9d\" (UniqueName: \"kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d\") pod \"data-science-collector-targetallocator-6bcc5b4468-zdqkh\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.340053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.340028 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.393873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.393850 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:32.393873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.393872 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrfxq\" (UniqueName: \"kubernetes.io/projected/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-kube-api-access-rrfxq\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:32.394050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.393882 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:32.394050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.393892 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:32.394050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.393901 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:32.394050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.393911 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:32.614723 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.614698 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh"] Oct 31 00:14:32.617285 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:32.617261 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2608c22b_ce28_4652_8aef_0c3038207b40.slice/crio-13d1feb3eb166ee7f3c6b413f4272fcae03e1601870962858fe80d92c0fc9057 WatchSource:0}: Error finding container 13d1feb3eb166ee7f3c6b413f4272fcae03e1601870962858fe80d92c0fc9057: Status 404 returned error can't find the container with id 13d1feb3eb166ee7f3c6b413f4272fcae03e1601870962858fe80d92c0fc9057 Oct 31 00:14:32.619365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.619336 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:32.619434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.619390 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:32.718777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.718738 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" event={"ID":"2608c22b-ce28-4652-8aef-0c3038207b40","Type":"ContainerStarted","Data":"20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656"} Oct 31 00:14:32.719006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.718818 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" event={"ID":"2608c22b-ce28-4652-8aef-0c3038207b40","Type":"ContainerStarted","Data":"13d1feb3eb166ee7f3c6b413f4272fcae03e1601870962858fe80d92c0fc9057"} Oct 31 00:14:32.719006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.718915 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:32.719868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.719837 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" podUID="2608c22b-ce28-4652-8aef-0c3038207b40" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.85:8080/readyz\": dial tcp 10.132.0.85:8080: connect: connection refused" Oct 31 00:14:32.720487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.720468 5073 generic.go:356] "Generic (PLEG): container finished" podID="aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" containerID="d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce" exitCode=0 Oct 31 00:14:32.720581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.720496 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8","Type":"ContainerDied","Data":"d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce"} Oct 31 00:14:32.720581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.720522 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.720581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.720544 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"aff51977-fa8e-4050-a2cc-9dfbcb2f63c8","Type":"ContainerDied","Data":"d9112990c429751c92bed1cd78f15818e1236738bc679d5ef7e5c7a082b12eb2"} Oct 31 00:14:32.720581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.720568 5073 scope.go:122] "RemoveContainer" containerID="d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce" Oct 31 00:14:32.730020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.730001 5073 scope.go:122] "RemoveContainer" containerID="d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce" Oct 31 00:14:32.730290 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:32.730271 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce\": container with ID starting with d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce not found: ID does not exist" containerID="d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce" Oct 31 00:14:32.730336 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.730296 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce"} err="failed to get container status \"d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce\": rpc error: code = NotFound desc = could not find container \"d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce\": container with ID starting with d4dd13f76eb9231f41d63470eb757552d060519f48750c63e46ab385e492f7ce not found: ID does not exist" Oct 31 00:14:32.739442 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.739398 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" podStartSLOduration=0.73938555 podStartE2EDuration="739.38555ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:32.738961122 +0000 UTC m=+956.799735817" watchObservedRunningTime="2026-10-31 00:14:32.73938555 +0000 UTC m=+956.800160244" Oct 31 00:14:32.754869 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.754843 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:32.761859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.761836 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:32.787546 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.787519 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:32.787935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.787920 5073 cpu_manager.go:414] "containerMap: removing container" podUID="aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" containerName="otc-container" Oct 31 00:14:32.787983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.787937 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" containerName="otc-container" Oct 31 00:14:32.788033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.788024 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" containerName="otc-container" Oct 31 00:14:32.791000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.790979 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.794922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.794902 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:32.795025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.795004 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:32.795577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.795554 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:32.795689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.795593 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:32.795689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.795598 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:32.795689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.795628 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:32.804188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.804171 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:32.899275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.899212 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.899275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.899258 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.899483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.899458 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xr2\" (UniqueName: \"kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.899574 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.899558 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.899634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.899588 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:32.899634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:32.899621 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.000812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.000760 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.001190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.000819 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.001190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.000857 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.001190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.000878 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.001190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.000914 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xr2\" (UniqueName: \"kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.001190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.000956 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.001429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.001406 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.003684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.003659 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.003777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.003726 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.004002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.003983 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.004065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.003991 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.013194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.013176 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xr2\" (UniqueName: \"kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2\") pod \"data-science-collector-collector-1\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.106905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.106876 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:33.383015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.382992 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:33.384953 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:33.384929 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae34ebf0_8777_4bf0_9b90_06b38c7d34e5.slice/crio-22b19b6ceecaed6c7c8b4b589a2d309795d0ff643883ddeada1f3b8d30030765 WatchSource:0}: Error finding container 22b19b6ceecaed6c7c8b4b589a2d309795d0ff643883ddeada1f3b8d30030765: Status 404 returned error can't find the container with id 22b19b6ceecaed6c7c8b4b589a2d309795d0ff643883ddeada1f3b8d30030765 Oct 31 00:14:33.387135 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.387100 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:33.387244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.387164 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:33.734078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.734037 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5","Type":"ContainerStarted","Data":"c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33"} Oct 31 00:14:33.734225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.734134 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5","Type":"ContainerStarted","Data":"22b19b6ceecaed6c7c8b4b589a2d309795d0ff643883ddeada1f3b8d30030765"} Oct 31 00:14:33.735287 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.735267 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:33.755078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.755037 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.75502518 podStartE2EDuration="1.75502518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:33.753450683 +0000 UTC m=+957.814225379" watchObservedRunningTime="2026-10-31 00:14:33.75502518 +0000 UTC m=+957.815799874" Oct 31 00:14:33.802777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.802747 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g"] Oct 31 00:14:33.802959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:33.802929 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" podUID="da7906d2-b772-4472-b5fd-09cdd5512a4c" containerName="ta-container" containerID="cri-o://e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92" gracePeriod=30 Oct 31 00:14:34.058882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.058857 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff51977-fa8e-4050-a2cc-9dfbcb2f63c8" path="/var/lib/kubelet/pods/aff51977-fa8e-4050-a2cc-9dfbcb2f63c8/volumes" Oct 31 00:14:34.060405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.060388 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:34.211388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.211355 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal\" (UniqueName: \"kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal\") pod \"da7906d2-b772-4472-b5fd-09cdd5512a4c\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " Oct 31 00:14:34.211522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.211491 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert\") pod \"da7906d2-b772-4472-b5fd-09cdd5512a4c\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " Oct 31 00:14:34.211593 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.211535 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j\" (UniqueName: \"kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j\") pod \"da7906d2-b772-4472-b5fd-09cdd5512a4c\" (UID: \"da7906d2-b772-4472-b5fd-09cdd5512a4c\") " Oct 31 00:14:34.211663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.211594 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal" pod "da7906d2-b772-4472-b5fd-09cdd5512a4c" (UID: "da7906d2-b772-4472-b5fd-09cdd5512a4c"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:34.211739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.211724 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/da7906d2-b772-4472-b5fd-09cdd5512a4c-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:34.213679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.213658 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j" pod "da7906d2-b772-4472-b5fd-09cdd5512a4c" (UID: "da7906d2-b772-4472-b5fd-09cdd5512a4c"). InnerVolumeSpecName "kube-api-access-jtm6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:34.213773 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.213728 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert" pod "da7906d2-b772-4472-b5fd-09cdd5512a4c" (UID: "da7906d2-b772-4472-b5fd-09cdd5512a4c"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:34.312224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.312169 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/da7906d2-b772-4472-b5fd-09cdd5512a4c-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:34.312224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.312194 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtm6j\" (UniqueName: \"kubernetes.io/projected/da7906d2-b772-4472-b5fd-09cdd5512a4c-kube-api-access-jtm6j\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:34.740288 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.740253 5073 generic.go:356] "Generic (PLEG): container finished" podID="da7906d2-b772-4472-b5fd-09cdd5512a4c" containerID="e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92" exitCode=0 Oct 31 00:14:34.740439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.740299 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" Oct 31 00:14:34.740439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.740330 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" event={"ID":"da7906d2-b772-4472-b5fd-09cdd5512a4c","Type":"ContainerDied","Data":"e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92"} Oct 31 00:14:34.740439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.740373 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g" event={"ID":"da7906d2-b772-4472-b5fd-09cdd5512a4c","Type":"ContainerDied","Data":"8a964dbe5e680f9302e24c1067fc8f21962f60a1f8b2edc90f89fcc9c6229b3f"} Oct 31 00:14:34.740439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.740389 5073 scope.go:122] "RemoveContainer" containerID="e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92" Oct 31 00:14:34.749654 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.749635 5073 scope.go:122] "RemoveContainer" containerID="e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92" Oct 31 00:14:34.750004 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:34.749983 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92\": container with ID starting with e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92 not found: ID does not exist" containerID="e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92" Oct 31 00:14:34.750095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.750011 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92"} err="failed to get container status \"e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92\": rpc error: code = NotFound desc = could not find container \"e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92\": container with ID starting with e3279eaf5dfe2c3081117fc5c41d8a395cc1e57827ec624f8733ebe976423c92 not found: ID does not exist" Oct 31 00:14:34.761686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.761664 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g"] Oct 31 00:14:34.765157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:34.765138 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-567d597c85-gw89g"] Oct 31 00:14:36.058473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:36.058446 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7906d2-b772-4472-b5fd-09cdd5512a4c" path="/var/lib/kubelet/pods/da7906d2-b772-4472-b5fd-09cdd5512a4c/volumes" Oct 31 00:14:37.691729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.691698 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9"] Oct 31 00:14:37.692131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.692054 5073 cpu_manager.go:414] "containerMap: removing container" podUID="da7906d2-b772-4472-b5fd-09cdd5512a4c" containerName="ta-container" Oct 31 00:14:37.692131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.692066 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7906d2-b772-4472-b5fd-09cdd5512a4c" containerName="ta-container" Oct 31 00:14:37.692205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.692166 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7906d2-b772-4472-b5fd-09cdd5512a4c" containerName="ta-container" Oct 31 00:14:37.696794 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.696767 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.705136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.705096 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9"] Oct 31 00:14:37.735693 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.735664 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgww9\" (UniqueName: \"kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.735841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.735699 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.735888 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.735850 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.836988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.836965 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgww9\" (UniqueName: \"kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.837126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.836994 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.837126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.837039 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.837653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.837632 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.839529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.839511 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:37.849334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:37.849307 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgww9\" (UniqueName: \"kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9\") pod \"data-science-collector-targetallocator-676fbdc8f8-lpzc9\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:38.009142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.009114 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:38.282914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.282888 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9"] Oct 31 00:14:38.285625 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:38.285593 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod823e842c_fb3e_416a_80a7_6765e1d8f72a.slice/crio-1cf105b284f6b7f42b5a6796f36efb627f1e291b7aaad7dd3283a4cb8730d545 WatchSource:0}: Error finding container 1cf105b284f6b7f42b5a6796f36efb627f1e291b7aaad7dd3283a4cb8730d545: Status 404 returned error can't find the container with id 1cf105b284f6b7f42b5a6796f36efb627f1e291b7aaad7dd3283a4cb8730d545 Oct 31 00:14:38.287926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.287889 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:38.288044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.287950 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:38.766581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.766541 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" event={"ID":"823e842c-fb3e-416a-80a7-6765e1d8f72a","Type":"ContainerStarted","Data":"8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23"} Oct 31 00:14:38.766581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.766582 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" event={"ID":"823e842c-fb3e-416a-80a7-6765e1d8f72a","Type":"ContainerStarted","Data":"1cf105b284f6b7f42b5a6796f36efb627f1e291b7aaad7dd3283a4cb8730d545"} Oct 31 00:14:38.767044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.766727 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:38.768653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.768625 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:38.786200 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.786157 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" podStartSLOduration=1.78614599 podStartE2EDuration="1.78614599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:38.784155429 +0000 UTC m=+962.844930116" watchObservedRunningTime="2026-10-31 00:14:38.78614599 +0000 UTC m=+962.846920685" Oct 31 00:14:38.843561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.843530 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh"] Oct 31 00:14:38.843765 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:38.843733 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" podUID="2608c22b-ce28-4652-8aef-0c3038207b40" containerName="ta-container" containerID="cri-o://20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656" gracePeriod=30 Oct 31 00:14:39.091310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.091284 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:39.148324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.148300 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal\" (UniqueName: \"kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal\") pod \"2608c22b-ce28-4652-8aef-0c3038207b40\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " Oct 31 00:14:39.148445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.148362 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert\") pod \"2608c22b-ce28-4652-8aef-0c3038207b40\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " Oct 31 00:14:39.148445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.148397 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d\" (UniqueName: \"kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d\") pod \"2608c22b-ce28-4652-8aef-0c3038207b40\" (UID: \"2608c22b-ce28-4652-8aef-0c3038207b40\") " Oct 31 00:14:39.148705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.148680 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal" pod "2608c22b-ce28-4652-8aef-0c3038207b40" (UID: "2608c22b-ce28-4652-8aef-0c3038207b40"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:39.150550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.150524 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert" pod "2608c22b-ce28-4652-8aef-0c3038207b40" (UID: "2608c22b-ce28-4652-8aef-0c3038207b40"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:39.150713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.150693 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d" pod "2608c22b-ce28-4652-8aef-0c3038207b40" (UID: "2608c22b-ce28-4652-8aef-0c3038207b40"). InnerVolumeSpecName "kube-api-access-t2m9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:39.249080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.249053 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2608c22b-ce28-4652-8aef-0c3038207b40-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:39.249080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.249078 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2608c22b-ce28-4652-8aef-0c3038207b40-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:39.249229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.249089 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2m9d\" (UniqueName: \"kubernetes.io/projected/2608c22b-ce28-4652-8aef-0c3038207b40-kube-api-access-t2m9d\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:39.772142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.772111 5073 generic.go:356] "Generic (PLEG): container finished" podID="2608c22b-ce28-4652-8aef-0c3038207b40" containerID="20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656" exitCode=0 Oct 31 00:14:39.772555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.772158 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" Oct 31 00:14:39.772555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.772203 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" event={"ID":"2608c22b-ce28-4652-8aef-0c3038207b40","Type":"ContainerDied","Data":"20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656"} Oct 31 00:14:39.772555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.772244 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh" event={"ID":"2608c22b-ce28-4652-8aef-0c3038207b40","Type":"ContainerDied","Data":"13d1feb3eb166ee7f3c6b413f4272fcae03e1601870962858fe80d92c0fc9057"} Oct 31 00:14:39.772555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.772260 5073 scope.go:122] "RemoveContainer" containerID="20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656" Oct 31 00:14:39.781322 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.781302 5073 scope.go:122] "RemoveContainer" containerID="20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656" Oct 31 00:14:39.781566 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:39.781547 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656\": container with ID starting with 20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656 not found: ID does not exist" containerID="20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656" Oct 31 00:14:39.781609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.781576 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656"} err="failed to get container status \"20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656\": rpc error: code = NotFound desc = could not find container \"20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656\": container with ID starting with 20b4d2ffae154d4293cdb15e1a7f7150603868e7c3d0392310cb7c5efe84b656 not found: ID does not exist" Oct 31 00:14:39.794146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.794120 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh"] Oct 31 00:14:39.799894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:39.799876 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-zdqkh"] Oct 31 00:14:40.058073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:40.058014 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2608c22b-ce28-4652-8aef-0c3038207b40" path="/var/lib/kubelet/pods/2608c22b-ce28-4652-8aef-0c3038207b40/volumes" Oct 31 00:14:43.085607 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:43.085574 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:14:43.345043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.344967 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h"] Oct 31 00:14:43.345324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.345311 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2608c22b-ce28-4652-8aef-0c3038207b40" containerName="ta-container" Oct 31 00:14:43.345371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.345325 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2608c22b-ce28-4652-8aef-0c3038207b40" containerName="ta-container" Oct 31 00:14:43.345403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.345394 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2608c22b-ce28-4652-8aef-0c3038207b40" containerName="ta-container" Oct 31 00:14:43.349696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.349677 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.360733 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.360709 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h"] Oct 31 00:14:43.379266 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.379234 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.379414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.379366 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qsd\" (UniqueName: \"kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.379478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.379426 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.479733 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.479711 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.479856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.479763 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qsd\" (UniqueName: \"kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.479856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.479803 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.480342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.480312 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.482317 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.482297 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.491967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.491946 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qsd\" (UniqueName: \"kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd\") pod \"data-science-collector-targetallocator-6bcc5b4468-wr24h\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.661038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.660982 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:43.936031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.935959 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h"] Oct 31 00:14:43.939425 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:43.939401 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf04afdcd_7f9f_4241_96de_538d002f035c.slice/crio-ce6646b2edb8e89ae7e38e2e43668dd24542a1a3beac9161794c35d36354025a WatchSource:0}: Error finding container ce6646b2edb8e89ae7e38e2e43668dd24542a1a3beac9161794c35d36354025a: Status 404 returned error can't find the container with id ce6646b2edb8e89ae7e38e2e43668dd24542a1a3beac9161794c35d36354025a Oct 31 00:14:43.941944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.941909 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:43.942038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:43.941979 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:44.797642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.797601 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" event={"ID":"f04afdcd-7f9f-4241-96de-538d002f035c","Type":"ContainerStarted","Data":"c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443"} Oct 31 00:14:44.798057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.797651 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" event={"ID":"f04afdcd-7f9f-4241-96de-538d002f035c","Type":"ContainerStarted","Data":"ce6646b2edb8e89ae7e38e2e43668dd24542a1a3beac9161794c35d36354025a"} Oct 31 00:14:44.798057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.797674 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:44.799381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.799359 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:44.818116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.818073 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" podStartSLOduration=1.818059842 podStartE2EDuration="1.818059842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:44.816094895 +0000 UTC m=+968.876869595" watchObservedRunningTime="2026-10-31 00:14:44.818059842 +0000 UTC m=+968.878834537" Oct 31 00:14:44.864707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.864677 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9"] Oct 31 00:14:44.864931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:44.864905 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" podUID="823e842c-fb3e-416a-80a7-6765e1d8f72a" containerName="ta-container" containerID="cri-o://8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23" gracePeriod=30 Oct 31 00:14:45.107922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.107901 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:45.193687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.193660 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9\" (UniqueName: \"kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9\") pod \"823e842c-fb3e-416a-80a7-6765e1d8f72a\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " Oct 31 00:14:45.193852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.193700 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert\") pod \"823e842c-fb3e-416a-80a7-6765e1d8f72a\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " Oct 31 00:14:45.193852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.193728 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal\" (UniqueName: \"kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal\") pod \"823e842c-fb3e-416a-80a7-6765e1d8f72a\" (UID: \"823e842c-fb3e-416a-80a7-6765e1d8f72a\") " Oct 31 00:14:45.194110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.194090 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal" pod "823e842c-fb3e-416a-80a7-6765e1d8f72a" (UID: "823e842c-fb3e-416a-80a7-6765e1d8f72a"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:45.195766 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.195744 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9" pod "823e842c-fb3e-416a-80a7-6765e1d8f72a" (UID: "823e842c-fb3e-416a-80a7-6765e1d8f72a"). InnerVolumeSpecName "kube-api-access-sgww9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:45.195870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.195833 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert" pod "823e842c-fb3e-416a-80a7-6765e1d8f72a" (UID: "823e842c-fb3e-416a-80a7-6765e1d8f72a"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:45.294689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.294663 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgww9\" (UniqueName: \"kubernetes.io/projected/823e842c-fb3e-416a-80a7-6765e1d8f72a-kube-api-access-sgww9\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:45.294689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.294686 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/823e842c-fb3e-416a-80a7-6765e1d8f72a-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:45.294843 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.294696 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/823e842c-fb3e-416a-80a7-6765e1d8f72a-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:45.803334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.803305 5073 generic.go:356] "Generic (PLEG): container finished" podID="823e842c-fb3e-416a-80a7-6765e1d8f72a" containerID="8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23" exitCode=0 Oct 31 00:14:45.803716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.803346 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" Oct 31 00:14:45.803716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.803382 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" event={"ID":"823e842c-fb3e-416a-80a7-6765e1d8f72a","Type":"ContainerDied","Data":"8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23"} Oct 31 00:14:45.803716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.803429 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9" event={"ID":"823e842c-fb3e-416a-80a7-6765e1d8f72a","Type":"ContainerDied","Data":"1cf105b284f6b7f42b5a6796f36efb627f1e291b7aaad7dd3283a4cb8730d545"} Oct 31 00:14:45.803716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.803447 5073 scope.go:122] "RemoveContainer" containerID="8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23" Oct 31 00:14:45.815293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.815272 5073 scope.go:122] "RemoveContainer" containerID="8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23" Oct 31 00:14:45.815849 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:45.815763 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23\": container with ID starting with 8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23 not found: ID does not exist" containerID="8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23" Oct 31 00:14:45.815849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.815824 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23"} err="failed to get container status \"8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23\": rpc error: code = NotFound desc = could not find container \"8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23\": container with ID starting with 8cc731ad06d9c87f399ef28de3068c545d9f801a26cc87944dc932ea3734ed23 not found: ID does not exist" Oct 31 00:14:45.828495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.828473 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9"] Oct 31 00:14:45.832638 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:45.832619 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-676fbdc8f8-lpzc9"] Oct 31 00:14:46.059348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:46.059293 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823e842c-fb3e-416a-80a7-6765e1d8f72a" path="/var/lib/kubelet/pods/823e842c-fb3e-416a-80a7-6765e1d8f72a/volumes" Oct 31 00:14:50.234950 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.234915 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:50.235500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.235149 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" containerName="otc-container" containerID="cri-o://c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33" gracePeriod=30 Oct 31 00:14:50.339301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.339267 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m"] Oct 31 00:14:50.339619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.339605 5073 cpu_manager.go:414] "containerMap: removing container" podUID="823e842c-fb3e-416a-80a7-6765e1d8f72a" containerName="ta-container" Oct 31 00:14:50.339619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.339619 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823e842c-fb3e-416a-80a7-6765e1d8f72a" containerName="ta-container" Oct 31 00:14:50.339774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.339696 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="823e842c-fb3e-416a-80a7-6765e1d8f72a" containerName="ta-container" Oct 31 00:14:50.342713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.342694 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.352812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.352755 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m"] Oct 31 00:14:50.429087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.429063 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.429237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.429119 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.429237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.429198 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkw5h\" (UniqueName: \"kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.530340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.530273 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.530340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.530321 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkw5h\" (UniqueName: \"kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.530490 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.530427 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.530979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.530959 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.533039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.533012 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.542666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.542645 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkw5h\" (UniqueName: \"kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h\") pod \"data-science-collector-targetallocator-54b7f486c8-qqq5m\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.657553 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.657530 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:50.931604 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.931575 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m"] Oct 31 00:14:50.934393 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:50.934367 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57881353_a2df_4fa5_b724_b48599926a74.slice/crio-abd801218650143eb6f59f82df96046923dc666a9c266b7d7bed0160cadc118d WatchSource:0}: Error finding container abd801218650143eb6f59f82df96046923dc666a9c266b7d7bed0160cadc118d: Status 404 returned error can't find the container with id abd801218650143eb6f59f82df96046923dc666a9c266b7d7bed0160cadc118d Oct 31 00:14:50.936917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.936885 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:50.936998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:50.936941 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:51.835166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.835130 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" event={"ID":"57881353-a2df-4fa5-b724-b48599926a74","Type":"ContainerStarted","Data":"41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9"} Oct 31 00:14:51.835624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.835174 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" event={"ID":"57881353-a2df-4fa5-b724-b48599926a74","Type":"ContainerStarted","Data":"abd801218650143eb6f59f82df96046923dc666a9c266b7d7bed0160cadc118d"} Oct 31 00:14:51.835624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.835290 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:51.837026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.837001 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:51.854981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.854928 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" podStartSLOduration=1.8549113560000001 podStartE2EDuration="1.854911356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:51.852916184 +0000 UTC m=+975.913690878" watchObservedRunningTime="2026-10-31 00:14:51.854911356 +0000 UTC m=+975.915686052" Oct 31 00:14:51.899015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.898980 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h"] Oct 31 00:14:51.899160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:51.899141 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" podUID="f04afdcd-7f9f-4241-96de-538d002f035c" containerName="ta-container" containerID="cri-o://c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443" gracePeriod=30 Oct 31 00:14:52.138927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.138899 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:52.245906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.245873 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal\" (UniqueName: \"kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal\") pod \"f04afdcd-7f9f-4241-96de-538d002f035c\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " Oct 31 00:14:52.246080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.245921 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert\") pod \"f04afdcd-7f9f-4241-96de-538d002f035c\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " Oct 31 00:14:52.246080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.245956 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd\" (UniqueName: \"kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd\") pod \"f04afdcd-7f9f-4241-96de-538d002f035c\" (UID: \"f04afdcd-7f9f-4241-96de-538d002f035c\") " Oct 31 00:14:52.246318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.246287 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal" pod "f04afdcd-7f9f-4241-96de-538d002f035c" (UID: "f04afdcd-7f9f-4241-96de-538d002f035c"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:52.248153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.248133 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert" pod "f04afdcd-7f9f-4241-96de-538d002f035c" (UID: "f04afdcd-7f9f-4241-96de-538d002f035c"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:52.248255 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.248241 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd" pod "f04afdcd-7f9f-4241-96de-538d002f035c" (UID: "f04afdcd-7f9f-4241-96de-538d002f035c"). InnerVolumeSpecName "kube-api-access-t7qsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:52.347188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.347164 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qsd\" (UniqueName: \"kubernetes.io/projected/f04afdcd-7f9f-4241-96de-538d002f035c-kube-api-access-t7qsd\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.347188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.347187 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f04afdcd-7f9f-4241-96de-538d002f035c-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.347323 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.347199 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f04afdcd-7f9f-4241-96de-538d002f035c-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.581615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.581593 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.649912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.649847 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal\") pod \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " Oct 31 00:14:52.649912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.649883 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs\") pod \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " Oct 31 00:14:52.650124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.649924 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2\" (UniqueName: \"kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2\") pod \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " Oct 31 00:14:52.650124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.649958 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs\") pod \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " Oct 31 00:14:52.650124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.650019 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert\") pod \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " Oct 31 00:14:52.650124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.650112 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs\") pod \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\" (UID: \"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5\") " Oct 31 00:14:52.650351 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.650318 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal" pod "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" (UID: "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:52.650444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.650428 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.652329 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.652299 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs" pod "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" (UID: "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:52.652454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.652332 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs" pod "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" (UID: "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:52.652454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.652391 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs" pod "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" (UID: "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:52.652740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.652714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2" pod "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" (UID: "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5"). InnerVolumeSpecName "kube-api-access-z8xr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:52.652740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.652714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert" pod "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" (UID: "ae34ebf0-8777-4bf0-9b90-06b38c7d34e5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:52.751246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.751216 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.751246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.751245 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xr2\" (UniqueName: \"kubernetes.io/projected/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-kube-api-access-z8xr2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.751432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.751258 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.751432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.751273 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.751432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.751287 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:52.842178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.842146 5073 generic.go:356] "Generic (PLEG): container finished" podID="ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" containerID="c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33" exitCode=0 Oct 31 00:14:52.842637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.842200 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.842637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.842243 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5","Type":"ContainerDied","Data":"c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33"} Oct 31 00:14:52.842637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.842310 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ae34ebf0-8777-4bf0-9b90-06b38c7d34e5","Type":"ContainerDied","Data":"22b19b6ceecaed6c7c8b4b589a2d309795d0ff643883ddeada1f3b8d30030765"} Oct 31 00:14:52.842637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.842348 5073 scope.go:122] "RemoveContainer" containerID="c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33" Oct 31 00:14:52.843660 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.843642 5073 generic.go:356] "Generic (PLEG): container finished" podID="f04afdcd-7f9f-4241-96de-538d002f035c" containerID="c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443" exitCode=0 Oct 31 00:14:52.843741 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.843691 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" Oct 31 00:14:52.843824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.843688 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" event={"ID":"f04afdcd-7f9f-4241-96de-538d002f035c","Type":"ContainerDied","Data":"c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443"} Oct 31 00:14:52.843870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.843825 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h" event={"ID":"f04afdcd-7f9f-4241-96de-538d002f035c","Type":"ContainerDied","Data":"ce6646b2edb8e89ae7e38e2e43668dd24542a1a3beac9161794c35d36354025a"} Oct 31 00:14:52.853251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.853225 5073 scope.go:122] "RemoveContainer" containerID="c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33" Oct 31 00:14:52.853662 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:52.853524 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33\": container with ID starting with c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33 not found: ID does not exist" containerID="c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33" Oct 31 00:14:52.853662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.853548 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33"} err="failed to get container status \"c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33\": rpc error: code = NotFound desc = could not find container \"c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33\": container with ID starting with c2e115562977f14c776a1745e05b2efb73e3d8683da9562f91f64f7211ebcb33 not found: ID does not exist" Oct 31 00:14:52.853662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.853564 5073 scope.go:122] "RemoveContainer" containerID="c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443" Oct 31 00:14:52.861302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.861288 5073 scope.go:122] "RemoveContainer" containerID="c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443" Oct 31 00:14:52.861541 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:52.861525 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443\": container with ID starting with c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443 not found: ID does not exist" containerID="c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443" Oct 31 00:14:52.861598 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.861546 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443"} err="failed to get container status \"c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443\": rpc error: code = NotFound desc = could not find container \"c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443\": container with ID starting with c12a814bfa7c778bea91f54612f2060d4c3459c71ef772a7cd4f5c87a36e4443 not found: ID does not exist" Oct 31 00:14:52.867225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.867207 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:52.871677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.871657 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:52.885131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.885111 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h"] Oct 31 00:14:52.891431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.891413 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6bcc5b4468-wr24h"] Oct 31 00:14:52.897643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.897624 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:52.897967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.897955 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f04afdcd-7f9f-4241-96de-538d002f035c" containerName="ta-container" Oct 31 00:14:52.898018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.897968 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04afdcd-7f9f-4241-96de-538d002f035c" containerName="ta-container" Oct 31 00:14:52.898018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.897982 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" containerName="otc-container" Oct 31 00:14:52.898018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.897986 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" containerName="otc-container" Oct 31 00:14:52.898108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.898062 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04afdcd-7f9f-4241-96de-538d002f035c" containerName="ta-container" Oct 31 00:14:52.898108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.898069 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" containerName="otc-container" Oct 31 00:14:52.902563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.902520 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.915232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.913675 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:52.928825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.928774 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:52.928915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.928884 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:52.928915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.928891 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:52.928985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.928941 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:52.929026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.929005 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:52.929090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.929079 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:52.952495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.952470 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.952602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.952499 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.952602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.952546 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvmbc\" (UniqueName: \"kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.952731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.952708 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.952837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.952817 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:52.952903 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:52.952857 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.053419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.053396 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvmbc\" (UniqueName: \"kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.053541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.053434 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.053541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.053510 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.053662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.053565 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.053662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.053597 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.053812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.053745 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.054511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.054485 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.056162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.056137 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.056241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.056167 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.056241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.056183 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.056309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.056248 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.065402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.065382 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvmbc\" (UniqueName: \"kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc\") pod \"data-science-collector-collector-1\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.218391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.218336 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:14:53.494035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.494005 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:14:53.496534 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:53.496507 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fba598_f508_4597_ba02_68e6a7013db3.slice/crio-acc79454473946984c62c308a2d94572dc32b23e8b092fac3b1f227fad71da1a WatchSource:0}: Error finding container acc79454473946984c62c308a2d94572dc32b23e8b092fac3b1f227fad71da1a: Status 404 returned error can't find the container with id acc79454473946984c62c308a2d94572dc32b23e8b092fac3b1f227fad71da1a Oct 31 00:14:53.498664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.498634 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:53.498729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.498690 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:53.850583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.850487 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02fba598-f508-4597-ba02-68e6a7013db3","Type":"ContainerStarted","Data":"f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0"} Oct 31 00:14:53.850583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.850546 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02fba598-f508-4597-ba02-68e6a7013db3","Type":"ContainerStarted","Data":"acc79454473946984c62c308a2d94572dc32b23e8b092fac3b1f227fad71da1a"} Oct 31 00:14:53.870359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:53.870190 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.870173968 podStartE2EDuration="1.870173968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:53.870152039 +0000 UTC m=+977.930926725" watchObservedRunningTime="2026-10-31 00:14:53.870173968 +0000 UTC m=+977.930948664" Oct 31 00:14:54.059193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:54.059161 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae34ebf0-8777-4bf0-9b90-06b38c7d34e5" path="/var/lib/kubelet/pods/ae34ebf0-8777-4bf0-9b90-06b38c7d34e5/volumes" Oct 31 00:14:54.059530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:54.059518 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04afdcd-7f9f-4241-96de-538d002f035c" path="/var/lib/kubelet/pods/f04afdcd-7f9f-4241-96de-538d002f035c/volumes" Oct 31 00:14:56.011157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.011126 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774"] Oct 31 00:14:56.015640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.015623 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.023778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.023750 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774"] Oct 31 00:14:56.079723 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.079699 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.079889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.079730 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf57f\" (UniqueName: \"kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.079889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.079767 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.181062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.181033 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.181223 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.181069 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf57f\" (UniqueName: \"kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.181223 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.181095 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.181685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.181662 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.183652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.183631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.193993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.193972 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf57f\" (UniqueName: \"kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f\") pod \"data-science-collector-targetallocator-b959f5bb8-rh774\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.329584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.329508 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.606215 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.606135 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774"] Oct 31 00:14:56.609239 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:14:56.609212 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda78fe6dd_29c3_4e4c_bdc3_e2edcac1cfe3.slice/crio-73b22153fc8e001448a23ae929014faf65a6c47206af76009b18f991f70ee7c8 WatchSource:0}: Error finding container 73b22153fc8e001448a23ae929014faf65a6c47206af76009b18f991f70ee7c8: Status 404 returned error can't find the container with id 73b22153fc8e001448a23ae929014faf65a6c47206af76009b18f991f70ee7c8 Oct 31 00:14:56.611559 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.611525 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:56.611656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.611582 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:14:56.875233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.875151 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" event={"ID":"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3","Type":"ContainerStarted","Data":"c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108"} Oct 31 00:14:56.875233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.875188 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" event={"ID":"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3","Type":"ContainerStarted","Data":"73b22153fc8e001448a23ae929014faf65a6c47206af76009b18f991f70ee7c8"} Oct 31 00:14:56.875443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.875353 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.877206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.877181 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:14:56.895686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.895633 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" podStartSLOduration=1.895617669 podStartE2EDuration="1.895617669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:56.895101187 +0000 UTC m=+980.955875886" watchObservedRunningTime="2026-10-31 00:14:56.895617669 +0000 UTC m=+980.956392369" Oct 31 00:14:56.941300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.941266 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m"] Oct 31 00:14:56.941498 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:56.941472 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" podUID="57881353-a2df-4fa5-b724-b48599926a74" containerName="ta-container" containerID="cri-o://41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9" gracePeriod=30 Oct 31 00:14:57.168831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.168808 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:57.190587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.190563 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal\" (UniqueName: \"kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal\") pod \"57881353-a2df-4fa5-b724-b48599926a74\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " Oct 31 00:14:57.190743 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.190595 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h\" (UniqueName: \"kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h\") pod \"57881353-a2df-4fa5-b724-b48599926a74\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " Oct 31 00:14:57.190743 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.190723 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert\") pod \"57881353-a2df-4fa5-b724-b48599926a74\" (UID: \"57881353-a2df-4fa5-b724-b48599926a74\") " Oct 31 00:14:57.190968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.190942 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal" pod "57881353-a2df-4fa5-b724-b48599926a74" (UID: "57881353-a2df-4fa5-b724-b48599926a74"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:57.191080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.191054 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/57881353-a2df-4fa5-b724-b48599926a74-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:57.193391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.193356 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h" pod "57881353-a2df-4fa5-b724-b48599926a74" (UID: "57881353-a2df-4fa5-b724-b48599926a74"). InnerVolumeSpecName "kube-api-access-kkw5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:57.194819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.194778 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert" pod "57881353-a2df-4fa5-b724-b48599926a74" (UID: "57881353-a2df-4fa5-b724-b48599926a74"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:57.291847 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.291817 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/57881353-a2df-4fa5-b724-b48599926a74-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:57.291948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.291847 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkw5h\" (UniqueName: \"kubernetes.io/projected/57881353-a2df-4fa5-b724-b48599926a74-kube-api-access-kkw5h\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:57.881563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.881535 5073 generic.go:356] "Generic (PLEG): container finished" podID="57881353-a2df-4fa5-b724-b48599926a74" containerID="41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9" exitCode=0 Oct 31 00:14:57.881711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.881579 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" Oct 31 00:14:57.881711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.881616 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" event={"ID":"57881353-a2df-4fa5-b724-b48599926a74","Type":"ContainerDied","Data":"41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9"} Oct 31 00:14:57.881711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.881657 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m" event={"ID":"57881353-a2df-4fa5-b724-b48599926a74","Type":"ContainerDied","Data":"abd801218650143eb6f59f82df96046923dc666a9c266b7d7bed0160cadc118d"} Oct 31 00:14:57.881711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.881674 5073 scope.go:122] "RemoveContainer" containerID="41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9" Oct 31 00:14:57.890740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.890723 5073 scope.go:122] "RemoveContainer" containerID="41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9" Oct 31 00:14:57.891045 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:14:57.891028 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9\": container with ID starting with 41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9 not found: ID does not exist" containerID="41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9" Oct 31 00:14:57.891102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.891054 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9"} err="failed to get container status \"41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9\": rpc error: code = NotFound desc = could not find container \"41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9\": container with ID starting with 41142b27ec35241038368ccfc4f08513961b855a9fcfb1c6b0ad5326e9cdfab9 not found: ID does not exist" Oct 31 00:14:57.902313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.902288 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m"] Oct 31 00:14:57.906119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:57.906099 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-qqq5m"] Oct 31 00:14:58.057847 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:14:58.057824 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57881353-a2df-4fa5-b724-b48599926a74" path="/var/lib/kubelet/pods/57881353-a2df-4fa5-b724-b48599926a74/volumes" Oct 31 00:15:13.085641 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:15:13.085607 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:15:43.085751 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:15:43.085717 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:16:13.085406 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:13.085336 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:16:19.935264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:19.935230 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t"] Oct 31 00:16:19.935645 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:19.935570 5073 cpu_manager.go:414] "containerMap: removing container" podUID="57881353-a2df-4fa5-b724-b48599926a74" containerName="ta-container" Oct 31 00:16:19.935645 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:19.935581 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57881353-a2df-4fa5-b724-b48599926a74" containerName="ta-container" Oct 31 00:16:19.935723 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:19.935654 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="57881353-a2df-4fa5-b724-b48599926a74" containerName="ta-container" Oct 31 00:16:19.938877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:19.938855 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:19.948485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:19.948449 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t"] Oct 31 00:16:20.017643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.017610 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.017813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.017646 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zt5\" (UniqueName: \"kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.017813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.017691 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.119013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.118986 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.119013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.119014 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zt5\" (UniqueName: \"kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.119217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.119135 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.119843 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.119754 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.121651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.121621 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.131528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.131502 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zt5\" (UniqueName: \"kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5\") pod \"data-science-collector-targetallocator-d94d6d4f9-88g9t\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.250846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.250823 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:20.522962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.522886 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t"] Oct 31 00:16:20.525911 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:16:20.525883 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod016a1784_513c_49ec_8d62_9d84d057b801.slice/crio-b76c3d6f1b12619ed2734753997ce0c7d247b375585e40a78ffb06cbeccae236 WatchSource:0}: Error finding container b76c3d6f1b12619ed2734753997ce0c7d247b375585e40a78ffb06cbeccae236: Status 404 returned error can't find the container with id b76c3d6f1b12619ed2734753997ce0c7d247b375585e40a78ffb06cbeccae236 Oct 31 00:16:20.528065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.528031 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:20.528132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:20.528087 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:21.255555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.255521 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" event={"ID":"016a1784-513c-49ec-8d62-9d84d057b801","Type":"ContainerStarted","Data":"5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea"} Oct 31 00:16:21.255555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.255562 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" event={"ID":"016a1784-513c-49ec-8d62-9d84d057b801","Type":"ContainerStarted","Data":"b76c3d6f1b12619ed2734753997ce0c7d247b375585e40a78ffb06cbeccae236"} Oct 31 00:16:21.255991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.255728 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:21.257541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.257521 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:21.293601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.293557 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" podStartSLOduration=2.293544652 podStartE2EDuration="2.293544652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:21.274143986 +0000 UTC m=+1065.334918681" watchObservedRunningTime="2026-10-31 00:16:21.293544652 +0000 UTC m=+1065.354319338" Oct 31 00:16:21.322031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.322005 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774"] Oct 31 00:16:21.322207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.322167 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" podUID="a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" containerName="ta-container" containerID="cri-o://c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108" gracePeriod=30 Oct 31 00:16:21.567628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.567605 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:16:21.631979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.631949 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal\" (UniqueName: \"kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal\") pod \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " Oct 31 00:16:21.632107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.632011 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert\") pod \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " Oct 31 00:16:21.632107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.632064 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f\" (UniqueName: \"kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f\") pod \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\" (UID: \"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3\") " Oct 31 00:16:21.632290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.632270 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal" pod "a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" (UID: "a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:21.634211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.634183 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert" pod "a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" (UID: "a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:21.634324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.634221 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f" pod "a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" (UID: "a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3"). InnerVolumeSpecName "kube-api-access-bf57f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:21.732550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.732524 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf57f\" (UniqueName: \"kubernetes.io/projected/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-kube-api-access-bf57f\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:21.732550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.732546 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:21.732693 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:21.732556 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:22.261716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.261684 5073 generic.go:356] "Generic (PLEG): container finished" podID="a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" containerID="c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108" exitCode=0 Oct 31 00:16:22.262171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.261726 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" Oct 31 00:16:22.262171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.261769 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" event={"ID":"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3","Type":"ContainerDied","Data":"c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108"} Oct 31 00:16:22.262171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.261832 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774" event={"ID":"a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3","Type":"ContainerDied","Data":"73b22153fc8e001448a23ae929014faf65a6c47206af76009b18f991f70ee7c8"} Oct 31 00:16:22.262171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.261849 5073 scope.go:122] "RemoveContainer" containerID="c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108" Oct 31 00:16:22.270185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.270169 5073 scope.go:122] "RemoveContainer" containerID="c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108" Oct 31 00:16:22.270466 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:22.270450 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108\": container with ID starting with c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108 not found: ID does not exist" containerID="c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108" Oct 31 00:16:22.270514 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.270476 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108"} err="failed to get container status \"c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108\": rpc error: code = NotFound desc = could not find container \"c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108\": container with ID starting with c588a2c834fa368dc93586ba040a69306a627b5af0cfb620ee83fa940c1c1108 not found: ID does not exist" Oct 31 00:16:22.279208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.279185 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774"] Oct 31 00:16:22.283670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:22.283651 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-b959f5bb8-rh774"] Oct 31 00:16:24.058302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:24.058276 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" path="/var/lib/kubelet/pods/a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3/volumes" Oct 31 00:16:25.128601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.128559 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h"] Oct 31 00:16:25.129115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.129095 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" containerName="ta-container" Oct 31 00:16:25.129175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.129117 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" containerName="ta-container" Oct 31 00:16:25.129258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.129247 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78fe6dd-29c3-4e4c-bdc3-e2edcac1cfe3" containerName="ta-container" Oct 31 00:16:25.133937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.133917 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.139385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.139362 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h"] Oct 31 00:16:25.160614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.160588 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.160728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.160618 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5h7r\" (UniqueName: \"kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.160728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.160709 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.261612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.261579 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.261612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.261613 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5h7r\" (UniqueName: \"kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.261824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.261644 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.262217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.262197 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.264081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.264060 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.274729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.274710 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5h7r\" (UniqueName: \"kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r\") pod \"data-science-collector-targetallocator-54b7f486c8-mbd2h\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.446825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.446740 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:25.721631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.721599 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h"] Oct 31 00:16:25.724232 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:16:25.724206 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce24e0c3_5c8a_4ff5_a29d_4905b4b94d90.slice/crio-aee5171e6181d1d4a57a71f66a135e7d27984cc97344eac04e727b15c8e2aef7 WatchSource:0}: Error finding container aee5171e6181d1d4a57a71f66a135e7d27984cc97344eac04e727b15c8e2aef7: Status 404 returned error can't find the container with id aee5171e6181d1d4a57a71f66a135e7d27984cc97344eac04e727b15c8e2aef7 Oct 31 00:16:25.726739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.726706 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:25.726815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:25.726759 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:26.283370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.283334 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" event={"ID":"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90","Type":"ContainerStarted","Data":"ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4"} Oct 31 00:16:26.283370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.283372 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" event={"ID":"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90","Type":"ContainerStarted","Data":"aee5171e6181d1d4a57a71f66a135e7d27984cc97344eac04e727b15c8e2aef7"} Oct 31 00:16:26.283856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.283476 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:26.285263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.285241 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:26.335164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.335118 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" podStartSLOduration=1.335106074 podStartE2EDuration="1.335106074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:26.306375714 +0000 UTC m=+1070.367150410" watchObservedRunningTime="2026-10-31 00:16:26.335106074 +0000 UTC m=+1070.395880838" Oct 31 00:16:26.386402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.386374 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t"] Oct 31 00:16:26.386562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.386514 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" podUID="016a1784-513c-49ec-8d62-9d84d057b801" containerName="ta-container" containerID="cri-o://5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea" gracePeriod=30 Oct 31 00:16:26.639600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.639577 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:26.673024 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.672998 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5\" (UniqueName: \"kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5\") pod \"016a1784-513c-49ec-8d62-9d84d057b801\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " Oct 31 00:16:26.673178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.673094 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal\" (UniqueName: \"kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal\") pod \"016a1784-513c-49ec-8d62-9d84d057b801\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " Oct 31 00:16:26.673178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.673152 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert\") pod \"016a1784-513c-49ec-8d62-9d84d057b801\" (UID: \"016a1784-513c-49ec-8d62-9d84d057b801\") " Oct 31 00:16:26.673484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.673459 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal" pod "016a1784-513c-49ec-8d62-9d84d057b801" (UID: "016a1784-513c-49ec-8d62-9d84d057b801"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:26.675306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.675274 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert" pod "016a1784-513c-49ec-8d62-9d84d057b801" (UID: "016a1784-513c-49ec-8d62-9d84d057b801"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:26.675396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.675381 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5" pod "016a1784-513c-49ec-8d62-9d84d057b801" (UID: "016a1784-513c-49ec-8d62-9d84d057b801"). InnerVolumeSpecName "kube-api-access-h7zt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:26.774188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.774164 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zt5\" (UniqueName: \"kubernetes.io/projected/016a1784-513c-49ec-8d62-9d84d057b801-kube-api-access-h7zt5\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:26.774293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.774191 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/016a1784-513c-49ec-8d62-9d84d057b801-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:26.774293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:26.774206 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/016a1784-513c-49ec-8d62-9d84d057b801-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:27.289421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.289360 5073 generic.go:356] "Generic (PLEG): container finished" podID="016a1784-513c-49ec-8d62-9d84d057b801" containerID="5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea" exitCode=0 Oct 31 00:16:27.289421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.289404 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" Oct 31 00:16:27.289825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.289452 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" event={"ID":"016a1784-513c-49ec-8d62-9d84d057b801","Type":"ContainerDied","Data":"5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea"} Oct 31 00:16:27.289825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.289485 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t" event={"ID":"016a1784-513c-49ec-8d62-9d84d057b801","Type":"ContainerDied","Data":"b76c3d6f1b12619ed2734753997ce0c7d247b375585e40a78ffb06cbeccae236"} Oct 31 00:16:27.289825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.289504 5073 scope.go:122] "RemoveContainer" containerID="5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea" Oct 31 00:16:27.301157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.299885 5073 scope.go:122] "RemoveContainer" containerID="5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea" Oct 31 00:16:27.301157 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:27.300200 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea\": container with ID starting with 5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea not found: ID does not exist" containerID="5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea" Oct 31 00:16:27.301157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.300229 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea"} err="failed to get container status \"5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea\": rpc error: code = NotFound desc = could not find container \"5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea\": container with ID starting with 5b258c2cc960a18bdb2500a9eed4553a72cb6d27c49c75379fba1265c827efea not found: ID does not exist" Oct 31 00:16:27.311505 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.311486 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t"] Oct 31 00:16:27.315587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:27.315567 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d94d6d4f9-88g9t"] Oct 31 00:16:28.058683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:28.058652 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016a1784-513c-49ec-8d62-9d84d057b801" path="/var/lib/kubelet/pods/016a1784-513c-49ec-8d62-9d84d057b801/volumes" Oct 31 00:16:30.464731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.464698 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw"] Oct 31 00:16:30.465133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.465051 5073 cpu_manager.go:414] "containerMap: removing container" podUID="016a1784-513c-49ec-8d62-9d84d057b801" containerName="ta-container" Oct 31 00:16:30.465133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.465063 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016a1784-513c-49ec-8d62-9d84d057b801" containerName="ta-container" Oct 31 00:16:30.465207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.465143 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="016a1784-513c-49ec-8d62-9d84d057b801" containerName="ta-container" Oct 31 00:16:30.469485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.469470 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.479073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.479044 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw"] Oct 31 00:16:30.503772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.503749 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.503908 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.503803 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plp49\" (UniqueName: \"kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.503908 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.503867 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.604950 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.604921 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.605081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.604955 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plp49\" (UniqueName: \"kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.605081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.604992 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.605558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.605538 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.607624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.607598 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.617627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.617604 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plp49\" (UniqueName: \"kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49\") pod \"data-science-collector-targetallocator-75cdff6cb-67jnw\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:30.781933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:30.781909 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:31.058360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.058333 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw"] Oct 31 00:16:31.061057 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:16:31.061028 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233f0f05_18d7_4ce6_a989_98f322d01d08.slice/crio-92246a09a80103039ce6ebd19310d080616d2321811f8cc739097fddc6870b04 WatchSource:0}: Error finding container 92246a09a80103039ce6ebd19310d080616d2321811f8cc739097fddc6870b04: Status 404 returned error can't find the container with id 92246a09a80103039ce6ebd19310d080616d2321811f8cc739097fddc6870b04 Oct 31 00:16:31.063588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.063557 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:31.063668 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.063612 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:31.309005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.308922 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" event={"ID":"233f0f05-18d7-4ce6-a989-98f322d01d08","Type":"ContainerStarted","Data":"c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461"} Oct 31 00:16:31.309005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.308973 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" event={"ID":"233f0f05-18d7-4ce6-a989-98f322d01d08","Type":"ContainerStarted","Data":"92246a09a80103039ce6ebd19310d080616d2321811f8cc739097fddc6870b04"} Oct 31 00:16:31.309222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.309067 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:31.310917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.310894 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:31.328227 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.328180 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" podStartSLOduration=1.328163256 podStartE2EDuration="1.328163256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:31.327994712 +0000 UTC m=+1075.388769408" watchObservedRunningTime="2026-10-31 00:16:31.328163256 +0000 UTC m=+1075.388937966" Oct 31 00:16:31.377300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.377275 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h"] Oct 31 00:16:31.377443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.377425 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" podUID="ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" containerName="ta-container" containerID="cri-o://ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4" gracePeriod=30 Oct 31 00:16:31.611143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.611120 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:31.714379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.714353 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal\" (UniqueName: \"kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal\") pod \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " Oct 31 00:16:31.714525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.714447 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert\") pod \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " Oct 31 00:16:31.714525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.714469 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r\" (UniqueName: \"kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r\") pod \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\" (UID: \"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90\") " Oct 31 00:16:31.714702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.714681 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal" pod "ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" (UID: "ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:31.716630 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.716603 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert" pod "ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" (UID: "ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:31.716734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.716644 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r" pod "ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" (UID: "ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90"). InnerVolumeSpecName "kube-api-access-v5h7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:31.815835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.815805 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.815935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.815836 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5h7r\" (UniqueName: \"kubernetes.io/projected/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-kube-api-access-v5h7r\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:31.815935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:31.815854 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:32.315044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.315011 5073 generic.go:356] "Generic (PLEG): container finished" podID="ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" containerID="ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4" exitCode=0 Oct 31 00:16:32.315180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.315054 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" Oct 31 00:16:32.315180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.315083 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" event={"ID":"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90","Type":"ContainerDied","Data":"ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4"} Oct 31 00:16:32.315180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.315112 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h" event={"ID":"ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90","Type":"ContainerDied","Data":"aee5171e6181d1d4a57a71f66a135e7d27984cc97344eac04e727b15c8e2aef7"} Oct 31 00:16:32.315180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.315128 5073 scope.go:122] "RemoveContainer" containerID="ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4" Oct 31 00:16:32.323616 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.323590 5073 scope.go:122] "RemoveContainer" containerID="ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4" Oct 31 00:16:32.323942 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:32.323924 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4\": container with ID starting with ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4 not found: ID does not exist" containerID="ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4" Oct 31 00:16:32.324002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.323952 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4"} err="failed to get container status \"ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4\": rpc error: code = NotFound desc = could not find container \"ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4\": container with ID starting with ac3b61565b67c7a9033c0c6a72d2486b656d0933c950768669d16bcc875918b4 not found: ID does not exist" Oct 31 00:16:32.332629 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.332609 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h"] Oct 31 00:16:32.339426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:32.339403 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54b7f486c8-mbd2h"] Oct 31 00:16:34.058164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:34.058135 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" path="/var/lib/kubelet/pods/ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90/volumes" Oct 31 00:16:43.086689 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:43.086652 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:16:47.691623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.691589 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:16:47.692238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.691821 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="02fba598-f508-4597-ba02-68e6a7013db3" containerName="otc-container" containerID="cri-o://f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0" gracePeriod=30 Oct 31 00:16:47.710816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.710767 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf"] Oct 31 00:16:47.711221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.711174 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" containerName="ta-container" Oct 31 00:16:47.711411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.711390 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" containerName="ta-container" Oct 31 00:16:47.711632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.711618 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce24e0c3-5c8a-4ff5-a29d-4905b4b94d90" containerName="ta-container" Oct 31 00:16:47.716348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.716327 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.726112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.726084 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf"] Oct 31 00:16:47.806024 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.805995 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf"] Oct 31 00:16:47.806367 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:47.806338 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data-science-collector-ta-server-cert kube-api-access-cvmpg ta-internal], unattached volumes=[], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" podUID="b7231670-0677-407c-a454-10347737fea3" Oct 31 00:16:47.836920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.836884 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.837090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.836932 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmpg\" (UniqueName: \"kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.837136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.837109 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.938060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.938004 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.938251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.938088 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.938251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.938137 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmpg\" (UniqueName: \"kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.938752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.938723 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.940674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.940649 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:47.950487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:47.950441 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmpg\" (UniqueName: \"kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg\") pod \"data-science-collector-targetallocator-559f5bd4b7-n6qmf\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:48.395978 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.395950 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:48.403168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.403148 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:48.442620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.442593 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg\" (UniqueName: \"kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg\") pod \"b7231670-0677-407c-a454-10347737fea3\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " Oct 31 00:16:48.442751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.442667 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal\") pod \"b7231670-0677-407c-a454-10347737fea3\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " Oct 31 00:16:48.442751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.442693 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert\") pod \"b7231670-0677-407c-a454-10347737fea3\" (UID: \"b7231670-0677-407c-a454-10347737fea3\") " Oct 31 00:16:48.443034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.443006 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal" pod "b7231670-0677-407c-a454-10347737fea3" (UID: "b7231670-0677-407c-a454-10347737fea3"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:48.444938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.444911 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert" pod "b7231670-0677-407c-a454-10347737fea3" (UID: "b7231670-0677-407c-a454-10347737fea3"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:48.445028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.444935 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg" pod "b7231670-0677-407c-a454-10347737fea3" (UID: "b7231670-0677-407c-a454-10347737fea3"). InnerVolumeSpecName "kube-api-access-cvmpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:48.543166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.543136 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b7231670-0677-407c-a454-10347737fea3-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:48.543166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.543166 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b7231670-0677-407c-a454-10347737fea3-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:48.543300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:48.543177 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvmpg\" (UniqueName: \"kubernetes.io/projected/b7231670-0677-407c-a454-10347737fea3-kube-api-access-cvmpg\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:49.400500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:49.400467 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf" Oct 31 00:16:49.434037 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:49.433999 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf"] Oct 31 00:16:49.439543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:49.439516 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-559f5bd4b7-n6qmf"] Oct 31 00:16:49.679133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:49.679064 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw"] Oct 31 00:16:49.679286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:49.679250 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" podUID="233f0f05-18d7-4ce6-a989-98f322d01d08" containerName="ta-container" containerID="cri-o://c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461" gracePeriod=30 Oct 31 00:16:49.735653 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:49.735619 5073 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233f0f05_18d7_4ce6_a989_98f322d01d08.slice/crio-conmon-c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:16:49.925258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:49.925232 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:50.052070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.052049 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.055944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.055920 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49\" (UniqueName: \"kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49\") pod \"233f0f05-18d7-4ce6-a989-98f322d01d08\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " Oct 31 00:16:50.056041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.055981 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert\") pod \"233f0f05-18d7-4ce6-a989-98f322d01d08\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " Oct 31 00:16:50.056112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.056044 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal\" (UniqueName: \"kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal\") pod \"233f0f05-18d7-4ce6-a989-98f322d01d08\" (UID: \"233f0f05-18d7-4ce6-a989-98f322d01d08\") " Oct 31 00:16:50.056418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.056390 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal" pod "233f0f05-18d7-4ce6-a989-98f322d01d08" (UID: "233f0f05-18d7-4ce6-a989-98f322d01d08"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:50.058023 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.057998 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7231670-0677-407c-a454-10347737fea3" path="/var/lib/kubelet/pods/b7231670-0677-407c-a454-10347737fea3/volumes" Oct 31 00:16:50.058107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.058081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49" pod "233f0f05-18d7-4ce6-a989-98f322d01d08" (UID: "233f0f05-18d7-4ce6-a989-98f322d01d08"). InnerVolumeSpecName "kube-api-access-plp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:50.058162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.058144 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert" pod "233f0f05-18d7-4ce6-a989-98f322d01d08" (UID: "233f0f05-18d7-4ce6-a989-98f322d01d08"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:50.157496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.157472 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs\") pod \"02fba598-f508-4597-ba02-68e6a7013db3\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " Oct 31 00:16:50.157634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.157503 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal\") pod \"02fba598-f508-4597-ba02-68e6a7013db3\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " Oct 31 00:16:50.157634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.157573 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs\") pod \"02fba598-f508-4597-ba02-68e6a7013db3\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " Oct 31 00:16:50.157634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.157601 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc\" (UniqueName: \"kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc\") pod \"02fba598-f508-4597-ba02-68e6a7013db3\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " Oct 31 00:16:50.157634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.157628 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs\") pod \"02fba598-f508-4597-ba02-68e6a7013db3\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " Oct 31 00:16:50.157883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.157776 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert\") pod \"02fba598-f508-4597-ba02-68e6a7013db3\" (UID: \"02fba598-f508-4597-ba02-68e6a7013db3\") " Oct 31 00:16:50.158099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.158064 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal" pod "02fba598-f508-4597-ba02-68e6a7013db3" (UID: "02fba598-f508-4597-ba02-68e6a7013db3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:50.158209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.158138 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/233f0f05-18d7-4ce6-a989-98f322d01d08-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.158209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.158163 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plp49\" (UniqueName: \"kubernetes.io/projected/233f0f05-18d7-4ce6-a989-98f322d01d08-kube-api-access-plp49\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.158209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.158182 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/233f0f05-18d7-4ce6-a989-98f322d01d08-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.159734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.159715 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs" pod "02fba598-f508-4597-ba02-68e6a7013db3" (UID: "02fba598-f508-4597-ba02-68e6a7013db3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:50.160258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.160237 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert" pod "02fba598-f508-4597-ba02-68e6a7013db3" (UID: "02fba598-f508-4597-ba02-68e6a7013db3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:50.160258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.160251 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc" pod "02fba598-f508-4597-ba02-68e6a7013db3" (UID: "02fba598-f508-4597-ba02-68e6a7013db3"). InnerVolumeSpecName "kube-api-access-bvmbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:50.160440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.160419 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs" pod "02fba598-f508-4597-ba02-68e6a7013db3" (UID: "02fba598-f508-4597-ba02-68e6a7013db3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:50.160440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.160434 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs" pod "02fba598-f508-4597-ba02-68e6a7013db3" (UID: "02fba598-f508-4597-ba02-68e6a7013db3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:50.259513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.259492 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.259513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.259513 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02fba598-f508-4597-ba02-68e6a7013db3-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.259655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.259522 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.259655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.259532 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvmbc\" (UniqueName: \"kubernetes.io/projected/02fba598-f508-4597-ba02-68e6a7013db3-kube-api-access-bvmbc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.259655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.259541 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.259655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.259551 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02fba598-f508-4597-ba02-68e6a7013db3-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:50.406434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.406407 5073 generic.go:356] "Generic (PLEG): container finished" podID="02fba598-f508-4597-ba02-68e6a7013db3" containerID="f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0" exitCode=0 Oct 31 00:16:50.406759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.406460 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.406759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.406488 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02fba598-f508-4597-ba02-68e6a7013db3","Type":"ContainerDied","Data":"f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0"} Oct 31 00:16:50.406759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.406538 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02fba598-f508-4597-ba02-68e6a7013db3","Type":"ContainerDied","Data":"acc79454473946984c62c308a2d94572dc32b23e8b092fac3b1f227fad71da1a"} Oct 31 00:16:50.406759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.406560 5073 scope.go:122] "RemoveContainer" containerID="f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0" Oct 31 00:16:50.408346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.408312 5073 generic.go:356] "Generic (PLEG): container finished" podID="233f0f05-18d7-4ce6-a989-98f322d01d08" containerID="c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461" exitCode=0 Oct 31 00:16:50.408346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.408338 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" event={"ID":"233f0f05-18d7-4ce6-a989-98f322d01d08","Type":"ContainerDied","Data":"c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461"} Oct 31 00:16:50.408482 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.408364 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" event={"ID":"233f0f05-18d7-4ce6-a989-98f322d01d08","Type":"ContainerDied","Data":"92246a09a80103039ce6ebd19310d080616d2321811f8cc739097fddc6870b04"} Oct 31 00:16:50.408482 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.408367 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw" Oct 31 00:16:50.416708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.416690 5073 scope.go:122] "RemoveContainer" containerID="f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0" Oct 31 00:16:50.417008 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:50.416984 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0\": container with ID starting with f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0 not found: ID does not exist" containerID="f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0" Oct 31 00:16:50.417086 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.417017 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0"} err="failed to get container status \"f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0\": rpc error: code = NotFound desc = could not find container \"f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0\": container with ID starting with f790229d48c043bf7fb90765791e411a6400a1bb279a14923c2bb20d45ad37f0 not found: ID does not exist" Oct 31 00:16:50.417086 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.417031 5073 scope.go:122] "RemoveContainer" containerID="c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461" Oct 31 00:16:50.426704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.426635 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw"] Oct 31 00:16:50.426943 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.426716 5073 scope.go:122] "RemoveContainer" containerID="c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461" Oct 31 00:16:50.427213 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:16:50.427190 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461\": container with ID starting with c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461 not found: ID does not exist" containerID="c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461" Oct 31 00:16:50.427281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.427220 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461"} err="failed to get container status \"c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461\": rpc error: code = NotFound desc = could not find container \"c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461\": container with ID starting with c8ff6d73a4e323a618ad835083703714921d6d8cc3316bc71bcc2cc73c678461 not found: ID does not exist" Oct 31 00:16:50.431511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.431491 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75cdff6cb-67jnw"] Oct 31 00:16:50.443248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.443229 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:16:50.447507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.447489 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:16:50.475013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.474991 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:16:50.475298 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.475287 5073 cpu_manager.go:414] "containerMap: removing container" podUID="233f0f05-18d7-4ce6-a989-98f322d01d08" containerName="ta-container" Oct 31 00:16:50.475419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.475300 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233f0f05-18d7-4ce6-a989-98f322d01d08" containerName="ta-container" Oct 31 00:16:50.475419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.475311 5073 cpu_manager.go:414] "containerMap: removing container" podUID="02fba598-f508-4597-ba02-68e6a7013db3" containerName="otc-container" Oct 31 00:16:50.475419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.475316 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fba598-f508-4597-ba02-68e6a7013db3" containerName="otc-container" Oct 31 00:16:50.475419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.475390 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="233f0f05-18d7-4ce6-a989-98f322d01d08" containerName="ta-container" Oct 31 00:16:50.475419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.475399 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fba598-f508-4597-ba02-68e6a7013db3" containerName="otc-container" Oct 31 00:16:50.479561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.479547 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.483435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.483418 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:16:50.483531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.483449 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:16:50.483531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.483507 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:16:50.484035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.484014 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:16:50.484154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.484100 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:16:50.484214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.484172 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-db539ca7\"" Oct 31 00:16:50.489119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.489088 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:16:50.562031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.561970 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.562031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.562010 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.562031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.562027 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.562213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.562153 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.562213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.562200 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c79p\" (UniqueName: \"kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.562291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.562268 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.663575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.663550 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.663680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.663591 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.663680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.663616 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.663680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.663660 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.663856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.663709 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c79p\" (UniqueName: \"kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.663856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.663749 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.664545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.664331 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.666253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.666218 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.666362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.666345 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.666426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.666364 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.666426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.666392 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.676157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.676136 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c79p\" (UniqueName: \"kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p\") pod \"data-science-collector-collector-1\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:50.793078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:50.793051 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:16:51.065563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:51.065531 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:16:51.068729 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:16:51.068702 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bc63c4_42f4_439b_a6aa_5088e4c6a1b6.slice/crio-4abfbd5580ad4d8f25d9eb08d31efe5b917580063ee2bf4c0686623b448ec637 WatchSource:0}: Error finding container 4abfbd5580ad4d8f25d9eb08d31efe5b917580063ee2bf4c0686623b448ec637: Status 404 returned error can't find the container with id 4abfbd5580ad4d8f25d9eb08d31efe5b917580063ee2bf4c0686623b448ec637 Oct 31 00:16:51.070807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:51.070749 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:51.070925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:51.070832 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:16:51.417300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:51.417222 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6","Type":"ContainerStarted","Data":"e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf"} Oct 31 00:16:51.417300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:51.417271 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6","Type":"ContainerStarted","Data":"4abfbd5580ad4d8f25d9eb08d31efe5b917580063ee2bf4c0686623b448ec637"} Oct 31 00:16:51.437937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:51.437892 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.437880142 podStartE2EDuration="1.437880142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:51.436774892 +0000 UTC m=+1095.497549588" watchObservedRunningTime="2026-10-31 00:16:51.437880142 +0000 UTC m=+1095.498654836" Oct 31 00:16:52.060105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:52.060072 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fba598-f508-4597-ba02-68e6a7013db3" path="/var/lib/kubelet/pods/02fba598-f508-4597-ba02-68e6a7013db3/volumes" Oct 31 00:16:52.060436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:16:52.060423 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233f0f05-18d7-4ce6-a989-98f322d01d08" path="/var/lib/kubelet/pods/233f0f05-18d7-4ce6-a989-98f322d01d08/volumes" Oct 31 00:17:13.085903 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:17:13.085869 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:17:43.086700 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:17:43.086665 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:18:13.085554 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:18:13.085522 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:18:36.176807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:18:36.176767 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:18:36.180146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:18:36.180123 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:18:36.183941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:18:36.183922 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:18:36.187689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:18:36.187673 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:18:36.407420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:18:36.407395 5073 scope.go:122] "RemoveContainer" containerID="856e5fd43c2edfa5f6cfb6fbcae6a907336384c29a0ca3ccf4f641a6664f33bc" Oct 31 00:18:43.087466 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:18:43.087425 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:19:12.492957 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.492741 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:12.493478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.493019 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" containerName="otc-container" containerID="cri-o://e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf" gracePeriod=30 Oct 31 00:19:12.565316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.565284 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6"] Oct 31 00:19:12.570286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.570262 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.577906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.577592 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-zpbg7\"" Oct 31 00:19:12.577906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.577767 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:19:12.577906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.577866 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:19:12.583317 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.583287 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6"] Oct 31 00:19:12.646651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.646623 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.646854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.646658 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.646936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.646846 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwx4k\" (UniqueName: \"kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.706771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.706740 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6"] Oct 31 00:19:12.707008 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:12.706986 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data-science-collector-ta-server-cert kube-api-access-zwx4k ta-internal], unattached volumes=[], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" podUID="b67db054-a7a6-474a-8a61-ac4af208ddda" Oct 31 00:19:12.747872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.747807 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.747872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.747843 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.748008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.747887 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwx4k\" (UniqueName: \"kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.748569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.748505 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.750368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.750348 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.758516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.758497 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb"] Oct 31 00:19:12.761980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.761965 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.764764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.764740 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwx4k\" (UniqueName: \"kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k\") pod \"data-science-collector-targetallocator-78fd9d45c-z5qm6\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:12.777895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.777874 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb"] Oct 31 00:19:12.849237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.849213 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dbr\" (UniqueName: \"kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.849354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.849246 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.849354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.849280 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.950710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.950685 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dbr\" (UniqueName: \"kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.950831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.950719 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.950831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.950752 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.951371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.951353 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.953165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.953147 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:12.963570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:12.963551 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dbr\" (UniqueName: \"kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr\") pod \"data-science-collector-targetallocator-cf46ff94-g5tjb\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:13.058505 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.058445 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:13.065400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.065382 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:13.083110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.083086 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:13.086251 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:13.086224 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:19:13.152573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.152542 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal\") pod \"b67db054-a7a6-474a-8a61-ac4af208ddda\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " Oct 31 00:19:13.152702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.152608 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert\") pod \"b67db054-a7a6-474a-8a61-ac4af208ddda\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " Oct 31 00:19:13.152702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.152674 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k\" (UniqueName: \"kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k\") pod \"b67db054-a7a6-474a-8a61-ac4af208ddda\" (UID: \"b67db054-a7a6-474a-8a61-ac4af208ddda\") " Oct 31 00:19:13.152987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.152958 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal" pod "b67db054-a7a6-474a-8a61-ac4af208ddda" (UID: "b67db054-a7a6-474a-8a61-ac4af208ddda"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:13.156009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.155414 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert" pod "b67db054-a7a6-474a-8a61-ac4af208ddda" (UID: "b67db054-a7a6-474a-8a61-ac4af208ddda"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:13.156330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.156309 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k" pod "b67db054-a7a6-474a-8a61-ac4af208ddda" (UID: "b67db054-a7a6-474a-8a61-ac4af208ddda"). InnerVolumeSpecName "kube-api-access-zwx4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:13.253423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.253400 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b67db054-a7a6-474a-8a61-ac4af208ddda-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:13.253423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.253422 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b67db054-a7a6-474a-8a61-ac4af208ddda-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:13.253558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.253433 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwx4k\" (UniqueName: \"kubernetes.io/projected/b67db054-a7a6-474a-8a61-ac4af208ddda-kube-api-access-zwx4k\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:13.359039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.358945 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb"] Oct 31 00:19:13.361649 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:19:13.361615 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7072e153_0c74_49e9_9684_4a2c781c3650.slice/crio-37d3d52e6466256dda6fdf0482a0262b74d80445109d98c6ba7d928e332ef181 WatchSource:0}: Error finding container 37d3d52e6466256dda6fdf0482a0262b74d80445109d98c6ba7d928e332ef181: Status 404 returned error can't find the container with id 37d3d52e6466256dda6fdf0482a0262b74d80445109d98c6ba7d928e332ef181 Oct 31 00:19:13.363690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.363658 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:13.363751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:13.363714 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:14.065429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.065344 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" event={"ID":"7072e153-0c74-49e9-9684-4a2c781c3650","Type":"ContainerStarted","Data":"3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761"} Oct 31 00:19:14.065429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.065369 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6" Oct 31 00:19:14.065429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.065389 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:14.065429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.065400 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" event={"ID":"7072e153-0c74-49e9-9684-4a2c781c3650","Type":"ContainerStarted","Data":"37d3d52e6466256dda6fdf0482a0262b74d80445109d98c6ba7d928e332ef181"} Oct 31 00:19:14.067517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.067498 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:14.085350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.085312 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" podStartSLOduration=2.085300743 podStartE2EDuration="2.085300743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:14.084076703 +0000 UTC m=+1238.144851398" watchObservedRunningTime="2026-10-31 00:19:14.085300743 +0000 UTC m=+1238.146075438" Oct 31 00:19:14.117446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.117423 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6"] Oct 31 00:19:14.126264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.126231 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-78fd9d45c-z5qm6"] Oct 31 00:19:14.839589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.839567 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:14.864902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.864867 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert\") pod \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " Oct 31 00:19:14.865038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.864933 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal\") pod \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " Oct 31 00:19:14.865038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.864962 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs\") pod \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " Oct 31 00:19:14.865038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.865012 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs\") pod \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " Oct 31 00:19:14.865208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.865092 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p\" (UniqueName: \"kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p\") pod \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " Oct 31 00:19:14.865208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.865150 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs\") pod \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\" (UID: \"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6\") " Oct 31 00:19:14.866257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.866224 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal" pod "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" (UID: "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:14.868536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.868509 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert" pod "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" (UID: "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:14.868731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.868710 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p" pod "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" (UID: "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6"). InnerVolumeSpecName "kube-api-access-7c79p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:14.869899 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.869878 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs" pod "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" (UID: "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:14.870000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.869899 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs" pod "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" (UID: "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:14.872121 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.872101 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs" pod "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" (UID: "15bc63c4-42f4-439b-a6aa-5088e4c6a1b6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:14.966521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.966460 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.966521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.966485 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c79p\" (UniqueName: \"kubernetes.io/projected/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-kube-api-access-7c79p\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.966521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.966499 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.966521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.966515 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.966719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.966530 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.966719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:14.966553 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:15.071981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.071956 5073 generic.go:356] "Generic (PLEG): container finished" podID="15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" containerID="e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf" exitCode=0 Oct 31 00:19:15.072412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.072003 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.072412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.072052 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6","Type":"ContainerDied","Data":"e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf"} Oct 31 00:19:15.072412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.072096 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"15bc63c4-42f4-439b-a6aa-5088e4c6a1b6","Type":"ContainerDied","Data":"4abfbd5580ad4d8f25d9eb08d31efe5b917580063ee2bf4c0686623b448ec637"} Oct 31 00:19:15.072412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.072117 5073 scope.go:122] "RemoveContainer" containerID="e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf" Oct 31 00:19:15.086970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.086952 5073 scope.go:122] "RemoveContainer" containerID="e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf" Oct 31 00:19:15.087245 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:15.087227 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf\": container with ID starting with e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf not found: ID does not exist" containerID="e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf" Oct 31 00:19:15.087288 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.087252 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf"} err="failed to get container status \"e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf\": rpc error: code = NotFound desc = could not find container \"e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf\": container with ID starting with e7c18c37d9da9a6959b71d6a7431a1e207be71a765a355248f00ee18b725c2cf not found: ID does not exist" Oct 31 00:19:15.101056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.101034 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:15.105590 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:15.105555 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:16.059500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:16.059467 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" path="/var/lib/kubelet/pods/15bc63c4-42f4-439b-a6aa-5088e4c6a1b6/volumes" Oct 31 00:19:16.059865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:16.059852 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67db054-a7a6-474a-8a61-ac4af208ddda" path="/var/lib/kubelet/pods/b67db054-a7a6-474a-8a61-ac4af208ddda/volumes" Oct 31 00:19:18.512899 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.512863 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9"] Oct 31 00:19:18.513389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.513204 5073 cpu_manager.go:414] "containerMap: removing container" podUID="15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" containerName="otc-container" Oct 31 00:19:18.513389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.513215 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" containerName="otc-container" Oct 31 00:19:18.513389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.513308 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bc63c4-42f4-439b-a6aa-5088e4c6a1b6" containerName="otc-container" Oct 31 00:19:18.517924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.517904 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.527332 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.527310 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9"] Oct 31 00:19:18.592601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.592573 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.592734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.592604 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhs4r\" (UniqueName: \"kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.592813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.592772 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.693282 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.693255 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.693404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.693290 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.693404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.693309 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhs4r\" (UniqueName: \"kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.694170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.694097 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.696018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.695992 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.706317 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.706289 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhs4r\" (UniqueName: \"kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r\") pod \"data-science-collector-targetallocator-5b77484b68-2z5f9\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:18.829671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:18.829611 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:19.114006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:19.113984 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9"] Oct 31 00:19:19.116875 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:19:19.116837 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda32bd036_0a5d_40c5_9c5d_7dec559f5840.slice/crio-e13a7e6ee3c815b67f3d8e7defa5a256c4db0a8eab343fa33361eb3e61b66efc WatchSource:0}: Error finding container e13a7e6ee3c815b67f3d8e7defa5a256c4db0a8eab343fa33361eb3e61b66efc: Status 404 returned error can't find the container with id e13a7e6ee3c815b67f3d8e7defa5a256c4db0a8eab343fa33361eb3e61b66efc Oct 31 00:19:19.119428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:19.119394 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:19.119533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:19.119462 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:20.095835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.095799 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" event={"ID":"a32bd036-0a5d-40c5-9c5d-7dec559f5840","Type":"ContainerStarted","Data":"010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101"} Oct 31 00:19:20.095835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.095835 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" event={"ID":"a32bd036-0a5d-40c5-9c5d-7dec559f5840","Type":"ContainerStarted","Data":"e13a7e6ee3c815b67f3d8e7defa5a256c4db0a8eab343fa33361eb3e61b66efc"} Oct 31 00:19:20.096286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.095850 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:20.097467 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.097445 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:20.117378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.117333 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" podStartSLOduration=2.11731915 podStartE2EDuration="2.11731915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:20.115526475 +0000 UTC m=+1244.176301180" watchObservedRunningTime="2026-10-31 00:19:20.11731915 +0000 UTC m=+1244.178093921" Oct 31 00:19:20.165508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.165486 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb"] Oct 31 00:19:20.165636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.165617 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" podUID="7072e153-0c74-49e9-9684-4a2c781c3650" containerName="ta-container" containerID="cri-o://3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761" gracePeriod=30 Oct 31 00:19:20.419407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.419385 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:20.510282 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.510241 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr\" (UniqueName: \"kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr\") pod \"7072e153-0c74-49e9-9684-4a2c781c3650\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " Oct 31 00:19:20.510437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.510321 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal\" (UniqueName: \"kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal\") pod \"7072e153-0c74-49e9-9684-4a2c781c3650\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " Oct 31 00:19:20.510437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.510408 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert\") pod \"7072e153-0c74-49e9-9684-4a2c781c3650\" (UID: \"7072e153-0c74-49e9-9684-4a2c781c3650\") " Oct 31 00:19:20.510716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.510689 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal" pod "7072e153-0c74-49e9-9684-4a2c781c3650" (UID: "7072e153-0c74-49e9-9684-4a2c781c3650"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:20.512631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.512597 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr" pod "7072e153-0c74-49e9-9684-4a2c781c3650" (UID: "7072e153-0c74-49e9-9684-4a2c781c3650"). InnerVolumeSpecName "kube-api-access-s9dbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:20.512631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.512610 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert" pod "7072e153-0c74-49e9-9684-4a2c781c3650" (UID: "7072e153-0c74-49e9-9684-4a2c781c3650"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:20.611542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.611515 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7072e153-0c74-49e9-9684-4a2c781c3650-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:20.611542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.611540 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7072e153-0c74-49e9-9684-4a2c781c3650-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:20.611698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:20.611551 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dbr\" (UniqueName: \"kubernetes.io/projected/7072e153-0c74-49e9-9684-4a2c781c3650-kube-api-access-s9dbr\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:21.101274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.101245 5073 generic.go:356] "Generic (PLEG): container finished" podID="7072e153-0c74-49e9-9684-4a2c781c3650" containerID="3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761" exitCode=0 Oct 31 00:19:21.101667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.101289 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" Oct 31 00:19:21.101667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.101327 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" event={"ID":"7072e153-0c74-49e9-9684-4a2c781c3650","Type":"ContainerDied","Data":"3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761"} Oct 31 00:19:21.101667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.101363 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb" event={"ID":"7072e153-0c74-49e9-9684-4a2c781c3650","Type":"ContainerDied","Data":"37d3d52e6466256dda6fdf0482a0262b74d80445109d98c6ba7d928e332ef181"} Oct 31 00:19:21.101667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.101379 5073 scope.go:122] "RemoveContainer" containerID="3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761" Oct 31 00:19:21.113615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.113596 5073 scope.go:122] "RemoveContainer" containerID="3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761" Oct 31 00:19:21.113899 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:21.113882 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761\": container with ID starting with 3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761 not found: ID does not exist" containerID="3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761" Oct 31 00:19:21.113961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.113906 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761"} err="failed to get container status \"3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761\": rpc error: code = NotFound desc = could not find container \"3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761\": container with ID starting with 3e4ca5a4ca97fedb753b06092f02fca96cbea78cf0a4c6c5df26663fd06d4761 not found: ID does not exist" Oct 31 00:19:21.127020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.126995 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb"] Oct 31 00:19:21.132018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:21.131999 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-cf46ff94-g5tjb"] Oct 31 00:19:22.058344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:22.058313 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7072e153-0c74-49e9-9684-4a2c781c3650" path="/var/lib/kubelet/pods/7072e153-0c74-49e9-9684-4a2c781c3650/volumes" Oct 31 00:19:35.914807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:35.914742 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh"] Oct 31 00:19:35.915366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:35.915342 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7072e153-0c74-49e9-9684-4a2c781c3650" containerName="ta-container" Oct 31 00:19:35.915409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:35.915370 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7072e153-0c74-49e9-9684-4a2c781c3650" containerName="ta-container" Oct 31 00:19:35.915491 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:35.915480 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7072e153-0c74-49e9-9684-4a2c781c3650" containerName="ta-container" Oct 31 00:19:35.919039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:35.919016 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:35.931653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:35.931612 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh"] Oct 31 00:19:36.036203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.036169 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gc6\" (UniqueName: \"kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.036352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.036228 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.036352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.036302 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.137259 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.137229 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.137400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.137266 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.137400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.137343 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gc6\" (UniqueName: \"kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.137931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.137910 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.139675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.139652 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.152049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.152027 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gc6\" (UniqueName: \"kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6\") pod \"data-science-collector-targetallocator-59846c9d57-8vqqh\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.236264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.236238 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:36.511251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.511117 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh"] Oct 31 00:19:36.514081 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:19:36.514055 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3031a1d3_4286_4029_80de_96e2407074ba.slice/crio-85a49bba234bd3d98958a714917f4980e53bb233a9b0055e427118c31a4eaeb4 WatchSource:0}: Error finding container 85a49bba234bd3d98958a714917f4980e53bb233a9b0055e427118c31a4eaeb4: Status 404 returned error can't find the container with id 85a49bba234bd3d98958a714917f4980e53bb233a9b0055e427118c31a4eaeb4 Oct 31 00:19:36.516462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.516429 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:36.516589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:36.516495 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:37.173075 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.173045 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" event={"ID":"3031a1d3-4286-4029-80de-96e2407074ba","Type":"ContainerStarted","Data":"e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841"} Oct 31 00:19:37.173435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.173081 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" event={"ID":"3031a1d3-4286-4029-80de-96e2407074ba","Type":"ContainerStarted","Data":"85a49bba234bd3d98958a714917f4980e53bb233a9b0055e427118c31a4eaeb4"} Oct 31 00:19:37.173435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.173195 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:37.174587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.174567 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:37.194630 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.194581 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" podStartSLOduration=2.194565511 podStartE2EDuration="2.194565511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:37.193654016 +0000 UTC m=+1261.254428722" watchObservedRunningTime="2026-10-31 00:19:37.194565511 +0000 UTC m=+1261.255340207" Oct 31 00:19:37.258693 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.258666 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9"] Oct 31 00:19:37.258874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.258850 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" podUID="a32bd036-0a5d-40c5-9c5d-7dec559f5840" containerName="ta-container" containerID="cri-o://010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101" gracePeriod=30 Oct 31 00:19:37.506551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.506525 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:37.653144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.653113 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert\") pod \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " Oct 31 00:19:37.653325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.653151 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r\" (UniqueName: \"kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r\") pod \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " Oct 31 00:19:37.653325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.653174 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal\" (UniqueName: \"kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal\") pod \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\" (UID: \"a32bd036-0a5d-40c5-9c5d-7dec559f5840\") " Oct 31 00:19:37.653549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.653527 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal" pod "a32bd036-0a5d-40c5-9c5d-7dec559f5840" (UID: "a32bd036-0a5d-40c5-9c5d-7dec559f5840"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:37.655344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.655322 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert" pod "a32bd036-0a5d-40c5-9c5d-7dec559f5840" (UID: "a32bd036-0a5d-40c5-9c5d-7dec559f5840"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.655489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.655472 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r" pod "a32bd036-0a5d-40c5-9c5d-7dec559f5840" (UID: "a32bd036-0a5d-40c5-9c5d-7dec559f5840"). InnerVolumeSpecName "kube-api-access-bhs4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:37.753844 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.753822 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a32bd036-0a5d-40c5-9c5d-7dec559f5840-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.753934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.753844 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhs4r\" (UniqueName: \"kubernetes.io/projected/a32bd036-0a5d-40c5-9c5d-7dec559f5840-kube-api-access-bhs4r\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.753934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:37.753855 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a32bd036-0a5d-40c5-9c5d-7dec559f5840-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:38.179545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.179514 5073 generic.go:356] "Generic (PLEG): container finished" podID="a32bd036-0a5d-40c5-9c5d-7dec559f5840" containerID="010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101" exitCode=0 Oct 31 00:19:38.179945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.179554 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" Oct 31 00:19:38.179945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.179581 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" event={"ID":"a32bd036-0a5d-40c5-9c5d-7dec559f5840","Type":"ContainerDied","Data":"010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101"} Oct 31 00:19:38.179945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.179609 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9" event={"ID":"a32bd036-0a5d-40c5-9c5d-7dec559f5840","Type":"ContainerDied","Data":"e13a7e6ee3c815b67f3d8e7defa5a256c4db0a8eab343fa33361eb3e61b66efc"} Oct 31 00:19:38.179945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.179624 5073 scope.go:122] "RemoveContainer" containerID="010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101" Oct 31 00:19:38.188064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.188047 5073 scope.go:122] "RemoveContainer" containerID="010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101" Oct 31 00:19:38.188335 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:38.188317 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101\": container with ID starting with 010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101 not found: ID does not exist" containerID="010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101" Oct 31 00:19:38.188397 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.188347 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101"} err="failed to get container status \"010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101\": rpc error: code = NotFound desc = could not find container \"010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101\": container with ID starting with 010bba24f121fa2f43f0911e69f14dbd8217e031f3bb88af0eb5f950b7510101 not found: ID does not exist" Oct 31 00:19:38.197266 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.197245 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9"] Oct 31 00:19:38.203809 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:38.203772 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b77484b68-2z5f9"] Oct 31 00:19:40.058929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:40.058896 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32bd036-0a5d-40c5-9c5d-7dec559f5840" path="/var/lib/kubelet/pods/a32bd036-0a5d-40c5-9c5d-7dec559f5840/volumes" Oct 31 00:19:41.656035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.655999 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:41.656554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.656533 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a32bd036-0a5d-40c5-9c5d-7dec559f5840" containerName="ta-container" Oct 31 00:19:41.656628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.656558 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32bd036-0a5d-40c5-9c5d-7dec559f5840" containerName="ta-container" Oct 31 00:19:41.656681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.656654 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32bd036-0a5d-40c5-9c5d-7dec559f5840" containerName="ta-container" Oct 31 00:19:41.661089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.661068 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.667543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.667521 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:19:41.667656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.667570 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:19:41.667656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.667627 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:19:41.668080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.668064 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:19:41.668650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.668619 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:19:41.668739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.668651 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:19:41.675333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.675311 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:41.788546 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.788520 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.788676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.788553 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.788676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.788583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.788676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.788659 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.788857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.788680 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44l87\" (UniqueName: \"kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.788857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.788721 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.889744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.889721 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l87\" (UniqueName: \"kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.889884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.889766 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.889884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.889829 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.889884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.889845 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.889884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.889868 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.890079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.889890 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.890581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.890547 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.892601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.892574 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.892825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.892804 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.892825 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.892818 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.892957 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.892809 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.902900 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.902876 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44l87\" (UniqueName: \"kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87\") pod \"data-science-collector-collector-0\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.973502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:41.973452 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:42.094520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.094491 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr"] Oct 31 00:19:42.098042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.098026 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.107761 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.107738 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr"] Oct 31 00:19:42.193352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.193325 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.193515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.193383 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.193515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.193461 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhsh\" (UniqueName: \"kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.250536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.250510 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:42.253472 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:19:42.253443 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e428fe_2b75_4d5b_bac0_2a9dd566ff32.slice/crio-1966a0d6afc824bcd69e1e35fda7cb79b94d0157118911f4e364abe9ffef2d60 WatchSource:0}: Error finding container 1966a0d6afc824bcd69e1e35fda7cb79b94d0157118911f4e364abe9ffef2d60: Status 404 returned error can't find the container with id 1966a0d6afc824bcd69e1e35fda7cb79b94d0157118911f4e364abe9ffef2d60 Oct 31 00:19:42.255555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.255521 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:42.255646 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.255579 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:42.294308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.294286 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.294430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.294337 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.294430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.294361 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhsh\" (UniqueName: \"kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.295135 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.294964 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.297082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.297064 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.306726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.306708 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhsh\" (UniqueName: \"kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh\") pod \"data-science-collector-targetallocator-6b94c7dcc8-jrtdr\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.410115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.410079 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:42.690888 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.690862 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr"] Oct 31 00:19:42.692895 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:19:42.692863 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod012c7418_617c_41dd_ac36_e82374d06d28.slice/crio-3df35d11f53be98ffe39ffd6ad2d7e2399424bb7e6b44161481e8ff7bc04ef7f WatchSource:0}: Error finding container 3df35d11f53be98ffe39ffd6ad2d7e2399424bb7e6b44161481e8ff7bc04ef7f: Status 404 returned error can't find the container with id 3df35d11f53be98ffe39ffd6ad2d7e2399424bb7e6b44161481e8ff7bc04ef7f Oct 31 00:19:42.695081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.695052 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:42.695173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:42.695107 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:19:43.085292 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:43.085258 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:19:43.210677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.210648 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" event={"ID":"012c7418-617c-41dd-ac36-e82374d06d28","Type":"ContainerStarted","Data":"fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f"} Oct 31 00:19:43.210862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.210685 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" event={"ID":"012c7418-617c-41dd-ac36-e82374d06d28","Type":"ContainerStarted","Data":"3df35d11f53be98ffe39ffd6ad2d7e2399424bb7e6b44161481e8ff7bc04ef7f"} Oct 31 00:19:43.210955 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.210930 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:43.212534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.212512 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:19:43.212636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.212540 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"26e428fe-2b75-4d5b-bac0-2a9dd566ff32","Type":"ContainerStarted","Data":"a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511"} Oct 31 00:19:43.212636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.212573 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"26e428fe-2b75-4d5b-bac0-2a9dd566ff32","Type":"ContainerStarted","Data":"1966a0d6afc824bcd69e1e35fda7cb79b94d0157118911f4e364abe9ffef2d60"} Oct 31 00:19:43.230602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.230555 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" podStartSLOduration=1.230540516 podStartE2EDuration="1.230540516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:43.229769417 +0000 UTC m=+1267.290544113" watchObservedRunningTime="2026-10-31 00:19:43.230540516 +0000 UTC m=+1267.291315201" Oct 31 00:19:43.250677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.250635 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.250620181 podStartE2EDuration="2.250620181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:43.249493168 +0000 UTC m=+1267.310267862" watchObservedRunningTime="2026-10-31 00:19:43.250620181 +0000 UTC m=+1267.311394877" Oct 31 00:19:43.309249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.309221 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh"] Oct 31 00:19:43.309471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.309446 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" podUID="3031a1d3-4286-4029-80de-96e2407074ba" containerName="ta-container" containerID="cri-o://e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841" gracePeriod=30 Oct 31 00:19:43.562487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.562461 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:43.712414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.712339 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6\" (UniqueName: \"kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6\") pod \"3031a1d3-4286-4029-80de-96e2407074ba\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " Oct 31 00:19:43.712414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.712373 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert\") pod \"3031a1d3-4286-4029-80de-96e2407074ba\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " Oct 31 00:19:43.712414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.712404 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal\" (UniqueName: \"kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal\") pod \"3031a1d3-4286-4029-80de-96e2407074ba\" (UID: \"3031a1d3-4286-4029-80de-96e2407074ba\") " Oct 31 00:19:43.712850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.712764 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal" pod "3031a1d3-4286-4029-80de-96e2407074ba" (UID: "3031a1d3-4286-4029-80de-96e2407074ba"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:43.714699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.714681 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert" pod "3031a1d3-4286-4029-80de-96e2407074ba" (UID: "3031a1d3-4286-4029-80de-96e2407074ba"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:43.714768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.714698 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6" pod "3031a1d3-4286-4029-80de-96e2407074ba" (UID: "3031a1d3-4286-4029-80de-96e2407074ba"). InnerVolumeSpecName "kube-api-access-s8gc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:43.813674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.813643 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gc6\" (UniqueName: \"kubernetes.io/projected/3031a1d3-4286-4029-80de-96e2407074ba-kube-api-access-s8gc6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:43.813674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.813671 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3031a1d3-4286-4029-80de-96e2407074ba-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:43.813832 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:43.813681 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3031a1d3-4286-4029-80de-96e2407074ba-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:44.218231 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.218203 5073 generic.go:356] "Generic (PLEG): container finished" podID="3031a1d3-4286-4029-80de-96e2407074ba" containerID="e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841" exitCode=0 Oct 31 00:19:44.218387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.218243 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" Oct 31 00:19:44.218387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.218263 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" event={"ID":"3031a1d3-4286-4029-80de-96e2407074ba","Type":"ContainerDied","Data":"e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841"} Oct 31 00:19:44.218387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.218306 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh" event={"ID":"3031a1d3-4286-4029-80de-96e2407074ba","Type":"ContainerDied","Data":"85a49bba234bd3d98958a714917f4980e53bb233a9b0055e427118c31a4eaeb4"} Oct 31 00:19:44.218387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.218323 5073 scope.go:122] "RemoveContainer" containerID="e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841" Oct 31 00:19:44.226907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.226888 5073 scope.go:122] "RemoveContainer" containerID="e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841" Oct 31 00:19:44.227148 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:19:44.227129 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841\": container with ID starting with e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841 not found: ID does not exist" containerID="e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841" Oct 31 00:19:44.227216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.227156 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841"} err="failed to get container status \"e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841\": rpc error: code = NotFound desc = could not find container \"e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841\": container with ID starting with e0e9d2634ae54f7d4119d94c4e60eb39225ae9ca4da3768c5295d26c34f60841 not found: ID does not exist" Oct 31 00:19:44.239870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.239845 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh"] Oct 31 00:19:44.243306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:44.243286 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-8vqqh"] Oct 31 00:19:46.060426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:19:46.060391 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3031a1d3-4286-4029-80de-96e2407074ba" path="/var/lib/kubelet/pods/3031a1d3-4286-4029-80de-96e2407074ba/volumes" Oct 31 00:20:13.086189 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:20:13.086153 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:20:42.397033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.396998 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 31 00:20:42.397436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.397330 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3031a1d3-4286-4029-80de-96e2407074ba" containerName="ta-container" Oct 31 00:20:42.397436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.397341 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3031a1d3-4286-4029-80de-96e2407074ba" containerName="ta-container" Oct 31 00:20:42.397436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.397429 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3031a1d3-4286-4029-80de-96e2407074ba" containerName="ta-container" Oct 31 00:20:42.401776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.401760 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:20:42.407268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.407245 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-otlp-identity\"/\"openshift-service-ca.crt\"" Oct 31 00:20:42.407268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.407259 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-otlp-identity\"/\"default-dockercfg-xwbq7\"" Oct 31 00:20:42.408424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.408396 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 31 00:20:42.449265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.449239 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca\") pod \"otlp-identity-spoof-probe\" (UID: \"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2\") " pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:20:42.550098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.550069 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca\") pod \"otlp-identity-spoof-probe\" (UID: \"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2\") " pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:20:42.550651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.550632 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca\") pod \"otlp-identity-spoof-probe\" (UID: \"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2\") " pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:20:42.714100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.714045 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:20:42.987405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:42.987371 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 31 00:20:42.990570 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:20:42.990537 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16d4509d_f5ef_41b1_bc8f_90c24e18f6b2.slice/crio-0fea078b9b70422496d7b8c53a85cbd0e16d308cc9a3988905f2ee9471dca571 WatchSource:0}: Error finding container 0fea078b9b70422496d7b8c53a85cbd0e16d308cc9a3988905f2ee9471dca571: Status 404 returned error can't find the container with id 0fea078b9b70422496d7b8c53a85cbd0e16d308cc9a3988905f2ee9471dca571 Oct 31 00:20:43.084528 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:20:43.084501 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:20:43.488208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:43.488168 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2","Type":"ContainerStarted","Data":"5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda"} Oct 31 00:20:43.488208 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:43.488208 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2","Type":"ContainerStarted","Data":"0fea078b9b70422496d7b8c53a85cbd0e16d308cc9a3988905f2ee9471dca571"} Oct 31 00:20:43.505707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:20:43.505663 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" podStartSLOduration=1.505650772 podStartE2EDuration="1.505650772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:43.504149717 +0000 UTC m=+1327.564924413" watchObservedRunningTime="2026-10-31 00:20:43.505650772 +0000 UTC m=+1327.566425466" Oct 31 00:21:13.085168 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:21:13.085135 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:21:43.085573 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:21:43.085538 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:22:13.084910 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:22:13.084838 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:22:43.084907 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:22:43.084870 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:22:52.658049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:22:52.658017 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 31 00:22:52.658498 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:22:52.658172 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" podUID="16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" containerName="otlp-spoof-probe" containerID="cri-o://5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda" gracePeriod=30 Oct 31 00:23:13.086336 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:13.086300 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:23:22.810615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:22.810593 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:23:22.975626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:22.975557 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca\") pod \"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2\" (UID: \"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2\") " Oct 31 00:23:22.975951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:22.975920 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca" pod "16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" (UID: "16d4509d-f5ef-41b1-bc8f-90c24e18f6b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:23.076572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.076539 5073 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2-service-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:23.200438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.200403 5073 generic.go:356] "Generic (PLEG): container finished" podID="16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" containerID="5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda" exitCode=137 Oct 31 00:23:23.200602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.200445 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" Oct 31 00:23:23.200602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.200439 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2","Type":"ContainerDied","Data":"5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda"} Oct 31 00:23:23.200602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.200554 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-otlp-identity/otlp-identity-spoof-probe" event={"ID":"16d4509d-f5ef-41b1-bc8f-90c24e18f6b2","Type":"ContainerDied","Data":"0fea078b9b70422496d7b8c53a85cbd0e16d308cc9a3988905f2ee9471dca571"} Oct 31 00:23:23.200602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.200571 5073 scope.go:122] "RemoveContainer" containerID="5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda" Oct 31 00:23:23.209573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.209558 5073 scope.go:122] "RemoveContainer" containerID="5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda" Oct 31 00:23:23.209849 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:23.209827 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda\": container with ID starting with 5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda not found: ID does not exist" containerID="5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda" Oct 31 00:23:23.209902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.209859 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda"} err="failed to get container status \"5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda\": rpc error: code = NotFound desc = could not find container \"5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda\": container with ID starting with 5bca4d24a99f4b1d30229b0922ff61b518f77049d60c25e0309cd621657c0dda not found: ID does not exist" Oct 31 00:23:23.222488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.222441 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 31 00:23:23.229080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:23.229010 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-otlp-identity/otlp-identity-spoof-probe"] Oct 31 00:23:24.058244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:24.058202 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" path="/var/lib/kubelet/pods/16d4509d-f5ef-41b1-bc8f-90c24e18f6b2/volumes" Oct 31 00:23:30.467084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.467050 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2"] Oct 31 00:23:30.467454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.467378 5073 cpu_manager.go:414] "containerMap: removing container" podUID="16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" containerName="otlp-spoof-probe" Oct 31 00:23:30.467454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.467388 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" containerName="otlp-spoof-probe" Oct 31 00:23:30.467524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.467466 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d4509d-f5ef-41b1-bc8f-90c24e18f6b2" containerName="otlp-spoof-probe" Oct 31 00:23:30.470523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.470500 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.480031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.480009 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2"] Oct 31 00:23:30.633519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.633493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfsb\" (UniqueName: \"kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.633519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.633529 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.633688 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.633648 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.734276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.734251 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfsb\" (UniqueName: \"kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.734386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.734287 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.734429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.734384 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.735063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.735041 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.736996 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.736969 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.749058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.749031 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfsb\" (UniqueName: \"kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb\") pod \"data-science-collector-targetallocator-5cf8957cff-wmlp2\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:30.783876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:30.783852 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:31.067356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.067287 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2"] Oct 31 00:23:31.070534 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:23:31.070506 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2893ed98_ac50_4093_9540_14a4d179f6ef.slice/crio-80cf313e02f1317df713c554fb336e4815b551b3213387fa7b6c5e66b3cd17d0 WatchSource:0}: Error finding container 80cf313e02f1317df713c554fb336e4815b551b3213387fa7b6c5e66b3cd17d0: Status 404 returned error can't find the container with id 80cf313e02f1317df713c554fb336e4815b551b3213387fa7b6c5e66b3cd17d0 Oct 31 00:23:31.072967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.072936 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:31.073046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.072995 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:31.239757 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.239726 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" event={"ID":"2893ed98-ac50-4093-9540-14a4d179f6ef","Type":"ContainerStarted","Data":"6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65"} Oct 31 00:23:31.239963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.239769 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:31.239963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.239797 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" event={"ID":"2893ed98-ac50-4093-9540-14a4d179f6ef","Type":"ContainerStarted","Data":"80cf313e02f1317df713c554fb336e4815b551b3213387fa7b6c5e66b3cd17d0"} Oct 31 00:23:31.240891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.240866 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" podUID="2893ed98-ac50-4093-9540-14a4d179f6ef" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.104:8080/readyz\": dial tcp 10.132.0.104:8080: connect: connection refused" Oct 31 00:23:31.264027 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:31.263981 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" podStartSLOduration=1.263966728 podStartE2EDuration="1.263966728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:31.263335607 +0000 UTC m=+1495.324110312" watchObservedRunningTime="2026-10-31 00:23:31.263966728 +0000 UTC m=+1495.324741424" Oct 31 00:23:32.245124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.245100 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:32.296803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.296752 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr"] Oct 31 00:23:32.296969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.296941 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" podUID="012c7418-617c-41dd-ac36-e82374d06d28" containerName="ta-container" containerID="cri-o://fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f" gracePeriod=30 Oct 31 00:23:32.537893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.537863 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:23:32.652309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.652278 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert\") pod \"012c7418-617c-41dd-ac36-e82374d06d28\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " Oct 31 00:23:32.652475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.652381 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal\" (UniqueName: \"kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal\") pod \"012c7418-617c-41dd-ac36-e82374d06d28\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " Oct 31 00:23:32.652475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.652417 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh\" (UniqueName: \"kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh\") pod \"012c7418-617c-41dd-ac36-e82374d06d28\" (UID: \"012c7418-617c-41dd-ac36-e82374d06d28\") " Oct 31 00:23:32.652752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.652725 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal" pod "012c7418-617c-41dd-ac36-e82374d06d28" (UID: "012c7418-617c-41dd-ac36-e82374d06d28"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:32.654610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.654592 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert" pod "012c7418-617c-41dd-ac36-e82374d06d28" (UID: "012c7418-617c-41dd-ac36-e82374d06d28"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:32.654684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.654644 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh" pod "012c7418-617c-41dd-ac36-e82374d06d28" (UID: "012c7418-617c-41dd-ac36-e82374d06d28"). InnerVolumeSpecName "kube-api-access-gkhsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:32.753607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.753578 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/012c7418-617c-41dd-ac36-e82374d06d28-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:32.753607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.753606 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkhsh\" (UniqueName: \"kubernetes.io/projected/012c7418-617c-41dd-ac36-e82374d06d28-kube-api-access-gkhsh\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:32.753735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:32.753621 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/012c7418-617c-41dd-ac36-e82374d06d28-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:33.251072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.251043 5073 generic.go:356] "Generic (PLEG): container finished" podID="012c7418-617c-41dd-ac36-e82374d06d28" containerID="fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f" exitCode=0 Oct 31 00:23:33.251513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.251100 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" Oct 31 00:23:33.251513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.251129 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" event={"ID":"012c7418-617c-41dd-ac36-e82374d06d28","Type":"ContainerDied","Data":"fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f"} Oct 31 00:23:33.251513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.251171 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr" event={"ID":"012c7418-617c-41dd-ac36-e82374d06d28","Type":"ContainerDied","Data":"3df35d11f53be98ffe39ffd6ad2d7e2399424bb7e6b44161481e8ff7bc04ef7f"} Oct 31 00:23:33.251513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.251188 5073 scope.go:122] "RemoveContainer" containerID="fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f" Oct 31 00:23:33.260639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.260621 5073 scope.go:122] "RemoveContainer" containerID="fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f" Oct 31 00:23:33.260930 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:33.260912 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f\": container with ID starting with fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f not found: ID does not exist" containerID="fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f" Oct 31 00:23:33.260977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.260937 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f"} err="failed to get container status \"fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f\": rpc error: code = NotFound desc = could not find container \"fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f\": container with ID starting with fe34bcb248acce2f5a826043b8578cd723e0b258ba09b4b11bdc390794a20d4f not found: ID does not exist" Oct 31 00:23:33.275876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.275856 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr"] Oct 31 00:23:33.280488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:33.280467 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b94c7dcc8-jrtdr"] Oct 31 00:23:34.059029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:34.058997 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012c7418-617c-41dd-ac36-e82374d06d28" path="/var/lib/kubelet/pods/012c7418-617c-41dd-ac36-e82374d06d28/volumes" Oct 31 00:23:35.991917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:35.991881 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm"] Oct 31 00:23:35.993418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:35.993391 5073 cpu_manager.go:414] "containerMap: removing container" podUID="012c7418-617c-41dd-ac36-e82374d06d28" containerName="ta-container" Oct 31 00:23:35.993585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:35.993570 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012c7418-617c-41dd-ac36-e82374d06d28" containerName="ta-container" Oct 31 00:23:35.993885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:35.993870 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="012c7418-617c-41dd-ac36-e82374d06d28" containerName="ta-container" Oct 31 00:23:35.999203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:35.999179 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.037912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.037891 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm"] Oct 31 00:23:36.073115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.073092 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tdd\" (UniqueName: \"kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.073230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.073128 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.073230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.073149 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.173534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.173508 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tdd\" (UniqueName: \"kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.173653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.173542 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.173653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.173563 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.174199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.174180 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.175988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.175964 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.189184 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.189097 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tdd\" (UniqueName: \"kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd\") pod \"data-science-collector-targetallocator-86b54c7c7b-rc5lm\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.206475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.205931 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:23:36.211162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.211145 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:23:36.212807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.212775 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:23:36.217923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.217904 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:23:36.310518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.310462 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:36.599552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.599528 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm"] Oct 31 00:23:36.601989 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:23:36.601961 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb911fc5a_3ca0_45a9_bf1a_5fa5b76805b9.slice/crio-7a626e14ef01ff8e7736e7a4db4e92878334e1fbf8ff64f978dd5af66d4d662a WatchSource:0}: Error finding container 7a626e14ef01ff8e7736e7a4db4e92878334e1fbf8ff64f978dd5af66d4d662a: Status 404 returned error can't find the container with id 7a626e14ef01ff8e7736e7a4db4e92878334e1fbf8ff64f978dd5af66d4d662a Oct 31 00:23:36.603973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.603937 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:36.604059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:36.603994 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:37.271152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.271112 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" event={"ID":"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9","Type":"ContainerStarted","Data":"d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612"} Oct 31 00:23:37.271531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.271164 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" event={"ID":"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9","Type":"ContainerStarted","Data":"7a626e14ef01ff8e7736e7a4db4e92878334e1fbf8ff64f978dd5af66d4d662a"} Oct 31 00:23:37.271531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.271183 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:37.273029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.273007 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:37.292334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.292287 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" podStartSLOduration=2.292270073 podStartE2EDuration="2.292270073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:37.290660106 +0000 UTC m=+1501.351434823" watchObservedRunningTime="2026-10-31 00:23:37.292270073 +0000 UTC m=+1501.353044770" Oct 31 00:23:37.340345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.340318 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2"] Oct 31 00:23:37.340471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.340450 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" podUID="2893ed98-ac50-4093-9540-14a4d179f6ef" containerName="ta-container" containerID="cri-o://6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65" gracePeriod=30 Oct 31 00:23:37.589230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.589208 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:37.685036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.685005 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal\" (UniqueName: \"kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal\") pod \"2893ed98-ac50-4093-9540-14a4d179f6ef\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " Oct 31 00:23:37.685166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.685046 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb\" (UniqueName: \"kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb\") pod \"2893ed98-ac50-4093-9540-14a4d179f6ef\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " Oct 31 00:23:37.685166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.685076 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert\") pod \"2893ed98-ac50-4093-9540-14a4d179f6ef\" (UID: \"2893ed98-ac50-4093-9540-14a4d179f6ef\") " Oct 31 00:23:37.685427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.685403 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal" pod "2893ed98-ac50-4093-9540-14a4d179f6ef" (UID: "2893ed98-ac50-4093-9540-14a4d179f6ef"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:37.687343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.687323 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert" pod "2893ed98-ac50-4093-9540-14a4d179f6ef" (UID: "2893ed98-ac50-4093-9540-14a4d179f6ef"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:37.687411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.687345 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb" pod "2893ed98-ac50-4093-9540-14a4d179f6ef" (UID: "2893ed98-ac50-4093-9540-14a4d179f6ef"). InnerVolumeSpecName "kube-api-access-fgfsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:37.785710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.785676 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2893ed98-ac50-4093-9540-14a4d179f6ef-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:37.785710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.785707 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfsb\" (UniqueName: \"kubernetes.io/projected/2893ed98-ac50-4093-9540-14a4d179f6ef-kube-api-access-fgfsb\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:37.785710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:37.785720 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2893ed98-ac50-4093-9540-14a4d179f6ef-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:38.276873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.276842 5073 generic.go:356] "Generic (PLEG): container finished" podID="2893ed98-ac50-4093-9540-14a4d179f6ef" containerID="6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65" exitCode=0 Oct 31 00:23:38.277273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.276891 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" Oct 31 00:23:38.277273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.276922 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" event={"ID":"2893ed98-ac50-4093-9540-14a4d179f6ef","Type":"ContainerDied","Data":"6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65"} Oct 31 00:23:38.277273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.276958 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2" event={"ID":"2893ed98-ac50-4093-9540-14a4d179f6ef","Type":"ContainerDied","Data":"80cf313e02f1317df713c554fb336e4815b551b3213387fa7b6c5e66b3cd17d0"} Oct 31 00:23:38.277273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.276974 5073 scope.go:122] "RemoveContainer" containerID="6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65" Oct 31 00:23:38.285696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.285678 5073 scope.go:122] "RemoveContainer" containerID="6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65" Oct 31 00:23:38.286097 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:38.286077 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65\": container with ID starting with 6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65 not found: ID does not exist" containerID="6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65" Oct 31 00:23:38.286149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.286106 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65"} err="failed to get container status \"6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65\": rpc error: code = NotFound desc = could not find container \"6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65\": container with ID starting with 6f9536ffbe9b57609cd0d14a14ebd2f4af7f8ac114454239d2625c8e28540a65 not found: ID does not exist" Oct 31 00:23:38.294472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.294450 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2"] Oct 31 00:23:38.296907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:38.296883 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5cf8957cff-wmlp2"] Oct 31 00:23:40.058411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:40.058377 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2893ed98-ac50-4093-9540-14a4d179f6ef" path="/var/lib/kubelet/pods/2893ed98-ac50-4093-9540-14a4d179f6ef/volumes" Oct 31 00:23:41.474760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.474728 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk"] Oct 31 00:23:41.475154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.475101 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2893ed98-ac50-4093-9540-14a4d179f6ef" containerName="ta-container" Oct 31 00:23:41.475154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.475114 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2893ed98-ac50-4093-9540-14a4d179f6ef" containerName="ta-container" Oct 31 00:23:41.475230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.475188 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2893ed98-ac50-4093-9540-14a4d179f6ef" containerName="ta-container" Oct 31 00:23:41.479597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.479579 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.491554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.491525 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk"] Oct 31 00:23:41.613381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.613357 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjn9\" (UniqueName: \"kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.613522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.613405 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.613522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.613486 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.714665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.714638 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjn9\" (UniqueName: \"kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.714774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.714686 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.714774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.714712 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.715267 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.715250 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.717288 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.717267 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.727249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.727191 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjn9\" (UniqueName: \"kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9\") pod \"data-science-collector-targetallocator-59846c9d57-7dcpk\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:41.791085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:41.791061 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:42.073233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.073210 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk"] Oct 31 00:23:42.074854 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:23:42.074829 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cdab2aa_8ce3_4347_b497_987fa5232939.slice/crio-6a59a29e68a2a51dc7fad07a7784ccf0f4300251ed7dd45b2eafbbc7c3c05b26 WatchSource:0}: Error finding container 6a59a29e68a2a51dc7fad07a7784ccf0f4300251ed7dd45b2eafbbc7c3c05b26: Status 404 returned error can't find the container with id 6a59a29e68a2a51dc7fad07a7784ccf0f4300251ed7dd45b2eafbbc7c3c05b26 Oct 31 00:23:42.076974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.076933 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:42.077060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.077010 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:42.299124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.299089 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" event={"ID":"4cdab2aa-8ce3-4347-b497-987fa5232939","Type":"ContainerStarted","Data":"7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67"} Oct 31 00:23:42.299124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.299132 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" event={"ID":"4cdab2aa-8ce3-4347-b497-987fa5232939","Type":"ContainerStarted","Data":"6a59a29e68a2a51dc7fad07a7784ccf0f4300251ed7dd45b2eafbbc7c3c05b26"} Oct 31 00:23:42.299360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.299276 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:42.301087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.301065 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:42.319429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.319389 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" podStartSLOduration=1.319377568 podStartE2EDuration="1.319377568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:42.317709392 +0000 UTC m=+1506.378484088" watchObservedRunningTime="2026-10-31 00:23:42.319377568 +0000 UTC m=+1506.380152324" Oct 31 00:23:42.368729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.368665 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm"] Oct 31 00:23:42.368862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.368843 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" podUID="b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" containerName="ta-container" containerID="cri-o://d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612" gracePeriod=30 Oct 31 00:23:42.612052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.612029 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:42.722925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.722844 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal\" (UniqueName: \"kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal\") pod \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " Oct 31 00:23:42.723087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.722931 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert\") pod \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " Oct 31 00:23:42.723087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.722982 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd\" (UniqueName: \"kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd\") pod \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\" (UID: \"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9\") " Oct 31 00:23:42.723264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.723241 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal" pod "b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" (UID: "b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:42.725238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.725209 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert" pod "b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" (UID: "b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:42.725358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.725260 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd" pod "b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" (UID: "b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9"). InnerVolumeSpecName "kube-api-access-k5tdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:42.823758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.823732 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tdd\" (UniqueName: \"kubernetes.io/projected/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-kube-api-access-k5tdd\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:42.823758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.823756 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:42.823913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:42.823768 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:43.087184 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:43.087123 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:23:43.305093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.305064 5073 generic.go:356] "Generic (PLEG): container finished" podID="b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" containerID="d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612" exitCode=0 Oct 31 00:23:43.305263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.305113 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" Oct 31 00:23:43.305263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.305155 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" event={"ID":"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9","Type":"ContainerDied","Data":"d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612"} Oct 31 00:23:43.305263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.305193 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm" event={"ID":"b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9","Type":"ContainerDied","Data":"7a626e14ef01ff8e7736e7a4db4e92878334e1fbf8ff64f978dd5af66d4d662a"} Oct 31 00:23:43.305263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.305211 5073 scope.go:122] "RemoveContainer" containerID="d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612" Oct 31 00:23:43.314219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.314205 5073 scope.go:122] "RemoveContainer" containerID="d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612" Oct 31 00:23:43.314462 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:43.314445 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612\": container with ID starting with d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612 not found: ID does not exist" containerID="d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612" Oct 31 00:23:43.314518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.314473 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612"} err="failed to get container status \"d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612\": rpc error: code = NotFound desc = could not find container \"d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612\": container with ID starting with d6785805aa99490fc1c9e74b9433988e95c8b63fa80ed3293a0e9b355dc74612 not found: ID does not exist" Oct 31 00:23:43.328414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.328392 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm"] Oct 31 00:23:43.330367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:43.330346 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-86b54c7c7b-rc5lm"] Oct 31 00:23:44.064775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:44.064738 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" path="/var/lib/kubelet/pods/b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9/volumes" Oct 31 00:23:53.521134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.521097 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb"] Oct 31 00:23:53.521739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.521718 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" containerName="ta-container" Oct 31 00:23:53.521739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.521735 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" containerName="ta-container" Oct 31 00:23:53.521930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.521838 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b911fc5a-3ca0-45a9-bf1a-5fa5b76805b9" containerName="ta-container" Oct 31 00:23:53.526106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.526090 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.537218 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.537193 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb"] Oct 31 00:23:53.601162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.601130 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.601307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.601178 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.601307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.601248 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthfx\" (UniqueName: \"kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.701650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.701618 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.701922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.701663 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.701922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.701687 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthfx\" (UniqueName: \"kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.702368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.702346 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.704522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.704499 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.714297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.714272 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthfx\" (UniqueName: \"kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx\") pod \"data-science-collector-targetallocator-f4b97b5f4-hccdb\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:53.838824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:53.838754 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:54.116320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.116205 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb"] Oct 31 00:23:54.119110 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:23:54.119083 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e63450e_d2d6_4a4f_bae9_4c59cbd9305d.slice/crio-97b08874a0cf3ad30714ea41f3b180ad9d3810cd2cf557b96fc12ee4cd469f6d WatchSource:0}: Error finding container 97b08874a0cf3ad30714ea41f3b180ad9d3810cd2cf557b96fc12ee4cd469f6d: Status 404 returned error can't find the container with id 97b08874a0cf3ad30714ea41f3b180ad9d3810cd2cf557b96fc12ee4cd469f6d Oct 31 00:23:54.120971 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.120938 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:54.121054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.120993 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:23:54.358530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.358493 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" event={"ID":"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d","Type":"ContainerStarted","Data":"88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07"} Oct 31 00:23:54.358694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.358538 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" event={"ID":"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d","Type":"ContainerStarted","Data":"97b08874a0cf3ad30714ea41f3b180ad9d3810cd2cf557b96fc12ee4cd469f6d"} Oct 31 00:23:54.358694 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.358668 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:54.360496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.360476 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:23:54.407995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.407907 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" podStartSLOduration=1.407890482 podStartE2EDuration="1.407890482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:54.407585319 +0000 UTC m=+1518.468360016" watchObservedRunningTime="2026-10-31 00:23:54.407890482 +0000 UTC m=+1518.468665178" Oct 31 00:23:54.496598 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.496560 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk"] Oct 31 00:23:54.496827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.496755 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" podUID="4cdab2aa-8ce3-4347-b497-987fa5232939" containerName="ta-container" containerID="cri-o://7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67" gracePeriod=30 Oct 31 00:23:54.745750 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.745722 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:54.808961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.808935 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9\" (UniqueName: \"kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9\") pod \"4cdab2aa-8ce3-4347-b497-987fa5232939\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " Oct 31 00:23:54.809119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.808971 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal\" (UniqueName: \"kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal\") pod \"4cdab2aa-8ce3-4347-b497-987fa5232939\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " Oct 31 00:23:54.809323 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.809303 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal" pod "4cdab2aa-8ce3-4347-b497-987fa5232939" (UID: "4cdab2aa-8ce3-4347-b497-987fa5232939"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:54.811063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.811040 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9" pod "4cdab2aa-8ce3-4347-b497-987fa5232939" (UID: "4cdab2aa-8ce3-4347-b497-987fa5232939"). InnerVolumeSpecName "kube-api-access-rtjn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:54.909813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.909750 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert\") pod \"4cdab2aa-8ce3-4347-b497-987fa5232939\" (UID: \"4cdab2aa-8ce3-4347-b497-987fa5232939\") " Oct 31 00:23:54.910003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.909990 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjn9\" (UniqueName: \"kubernetes.io/projected/4cdab2aa-8ce3-4347-b497-987fa5232939-kube-api-access-rtjn9\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:54.910049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.910005 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4cdab2aa-8ce3-4347-b497-987fa5232939-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:54.911796 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:54.911768 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert" pod "4cdab2aa-8ce3-4347-b497-987fa5232939" (UID: "4cdab2aa-8ce3-4347-b497-987fa5232939"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:55.010860 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.010838 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4cdab2aa-8ce3-4347-b497-987fa5232939-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:55.364867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.364836 5073 generic.go:356] "Generic (PLEG): container finished" podID="4cdab2aa-8ce3-4347-b497-987fa5232939" containerID="7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67" exitCode=0 Oct 31 00:23:55.365019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.364881 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" Oct 31 00:23:55.365019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.364921 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" event={"ID":"4cdab2aa-8ce3-4347-b497-987fa5232939","Type":"ContainerDied","Data":"7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67"} Oct 31 00:23:55.365019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.364964 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk" event={"ID":"4cdab2aa-8ce3-4347-b497-987fa5232939","Type":"ContainerDied","Data":"6a59a29e68a2a51dc7fad07a7784ccf0f4300251ed7dd45b2eafbbc7c3c05b26"} Oct 31 00:23:55.365019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.364983 5073 scope.go:122] "RemoveContainer" containerID="7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67" Oct 31 00:23:55.374354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.374339 5073 scope.go:122] "RemoveContainer" containerID="7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67" Oct 31 00:23:55.374621 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:23:55.374603 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67\": container with ID starting with 7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67 not found: ID does not exist" containerID="7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67" Oct 31 00:23:55.374673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.374629 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67"} err="failed to get container status \"7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67\": rpc error: code = NotFound desc = could not find container \"7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67\": container with ID starting with 7e8b31897f94caa18740e16c3d70622a6a186d6eda83ad70775792db05697e67 not found: ID does not exist" Oct 31 00:23:55.392528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.392504 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk"] Oct 31 00:23:55.397884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:55.397864 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-59846c9d57-7dcpk"] Oct 31 00:23:56.058839 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.058807 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cdab2aa-8ce3-4347-b497-987fa5232939" path="/var/lib/kubelet/pods/4cdab2aa-8ce3-4347-b497-987fa5232939/volumes" Oct 31 00:23:56.757653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.757621 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 31 00:23:56.758060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.758046 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4cdab2aa-8ce3-4347-b497-987fa5232939" containerName="ta-container" Oct 31 00:23:56.758112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.758062 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdab2aa-8ce3-4347-b497-987fa5232939" containerName="ta-container" Oct 31 00:23:56.758147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.758139 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cdab2aa-8ce3-4347-b497-987fa5232939" containerName="ta-container" Oct 31 00:23:56.762550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.762535 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:56.767094 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.767061 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"odh-observability-metrics-ca\"" Oct 31 00:23:56.769632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.769550 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 31 00:23:56.823543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.823520 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpskv\" (UniqueName: \"kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv\") pod \"operator-metrics-auth-probe\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:56.823667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.823558 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca\") pod \"operator-metrics-auth-probe\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:56.924439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.924414 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpskv\" (UniqueName: \"kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv\") pod \"operator-metrics-auth-probe\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:56.924562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.924451 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca\") pod \"operator-metrics-auth-probe\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:56.925043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.925027 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca\") pod \"operator-metrics-auth-probe\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:56.937346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:56.937327 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpskv\" (UniqueName: \"kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv\") pod \"operator-metrics-auth-probe\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:57.074842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:57.074773 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:23:57.346726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:57.346632 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 31 00:23:57.349591 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:23:57.349562 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d04f4b8_37a7_4d91_8806_ad361b32d1c0.slice/crio-10c00d000875957c531645e01877ad06c8ab773b328a0bff649bd2da145592b3 WatchSource:0}: Error finding container 10c00d000875957c531645e01877ad06c8ab773b328a0bff649bd2da145592b3: Status 404 returned error can't find the container with id 10c00d000875957c531645e01877ad06c8ab773b328a0bff649bd2da145592b3 Oct 31 00:23:57.375749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:57.375720 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"4d04f4b8-37a7-4d91-8806-ad361b32d1c0","Type":"ContainerStarted","Data":"10c00d000875957c531645e01877ad06c8ab773b328a0bff649bd2da145592b3"} Oct 31 00:23:57.497058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:57.497025 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:23:57.497237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:57.497215 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="26e428fe-2b75-4d5b-bac0-2a9dd566ff32" containerName="otc-container" containerID="cri-o://a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511" gracePeriod=30 Oct 31 00:23:58.381374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:58.381338 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"4d04f4b8-37a7-4d91-8806-ad361b32d1c0","Type":"ContainerStarted","Data":"689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a"} Oct 31 00:23:58.404403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:58.404359 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/operator-metrics-auth-probe" podStartSLOduration=2.404345433 podStartE2EDuration="2.404345433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:58.402572223 +0000 UTC m=+1522.463346919" watchObservedRunningTime="2026-10-31 00:23:58.404345433 +0000 UTC m=+1522.465120154" Oct 31 00:23:59.841513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.841489 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:23:59.845587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.845555 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs\") pod \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " Oct 31 00:23:59.845689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.845622 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87\" (UniqueName: \"kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87\") pod \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " Oct 31 00:23:59.845742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.845690 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs\") pod \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " Oct 31 00:23:59.845742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.845726 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal\" (UniqueName: \"kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal\") pod \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " Oct 31 00:23:59.845870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.845765 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert\") pod \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " Oct 31 00:23:59.845870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.845841 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs\") pod \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\" (UID: \"26e428fe-2b75-4d5b-bac0-2a9dd566ff32\") " Oct 31 00:23:59.846167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.846137 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal" pod "26e428fe-2b75-4d5b-bac0-2a9dd566ff32" (UID: "26e428fe-2b75-4d5b-bac0-2a9dd566ff32"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:59.847831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.847804 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs" pod "26e428fe-2b75-4d5b-bac0-2a9dd566ff32" (UID: "26e428fe-2b75-4d5b-bac0-2a9dd566ff32"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:59.847970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.847954 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs" pod "26e428fe-2b75-4d5b-bac0-2a9dd566ff32" (UID: "26e428fe-2b75-4d5b-bac0-2a9dd566ff32"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:59.848387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.848357 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs" pod "26e428fe-2b75-4d5b-bac0-2a9dd566ff32" (UID: "26e428fe-2b75-4d5b-bac0-2a9dd566ff32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:59.848501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.848450 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87" pod "26e428fe-2b75-4d5b-bac0-2a9dd566ff32" (UID: "26e428fe-2b75-4d5b-bac0-2a9dd566ff32"). InnerVolumeSpecName "kube-api-access-44l87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:59.848501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.848463 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert" pod "26e428fe-2b75-4d5b-bac0-2a9dd566ff32" (UID: "26e428fe-2b75-4d5b-bac0-2a9dd566ff32"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:59.947471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.947408 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:59.947471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.947433 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:59.947471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.947443 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44l87\" (UniqueName: \"kubernetes.io/projected/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-kube-api-access-44l87\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:59.947471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.947453 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:59.947471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.947461 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:59.947471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:23:59.947471 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26e428fe-2b75-4d5b-bac0-2a9dd566ff32-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:00.391745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.391712 5073 generic.go:356] "Generic (PLEG): container finished" podID="26e428fe-2b75-4d5b-bac0-2a9dd566ff32" containerID="a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511" exitCode=0 Oct 31 00:24:00.391913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.391757 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.391913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.391765 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"26e428fe-2b75-4d5b-bac0-2a9dd566ff32","Type":"ContainerDied","Data":"a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511"} Oct 31 00:24:00.391913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.391827 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"26e428fe-2b75-4d5b-bac0-2a9dd566ff32","Type":"ContainerDied","Data":"1966a0d6afc824bcd69e1e35fda7cb79b94d0157118911f4e364abe9ffef2d60"} Oct 31 00:24:00.391913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.391849 5073 scope.go:122] "RemoveContainer" containerID="a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511" Oct 31 00:24:00.400770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.400753 5073 scope.go:122] "RemoveContainer" containerID="a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511" Oct 31 00:24:00.401119 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:24:00.401101 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511\": container with ID starting with a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511 not found: ID does not exist" containerID="a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511" Oct 31 00:24:00.401176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.401126 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511"} err="failed to get container status \"a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511\": rpc error: code = NotFound desc = could not find container \"a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511\": container with ID starting with a5bc67481ab8e373c427b9e910da9ce22d7bea75f1689a881bf780d68e00e511 not found: ID does not exist" Oct 31 00:24:00.409382 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.409360 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:24:00.415667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:00.415645 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:24:02.059718 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:02.059684 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26e428fe-2b75-4d5b-bac0-2a9dd566ff32" path="/var/lib/kubelet/pods/26e428fe-2b75-4d5b-bac0-2a9dd566ff32/volumes" Oct 31 00:24:02.717677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:02.717644 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 31 00:24:02.717885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:02.717822 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/operator-metrics-auth-probe" podUID="4d04f4b8-37a7-4d91-8806-ad361b32d1c0" containerName="curl" containerID="cri-o://689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a" gracePeriod=30 Oct 31 00:24:13.085382 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:24:13.085347 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:24:25.572471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.572434 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f"] Oct 31 00:24:25.572976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.572767 5073 cpu_manager.go:414] "containerMap: removing container" podUID="26e428fe-2b75-4d5b-bac0-2a9dd566ff32" containerName="otc-container" Oct 31 00:24:25.572976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.572779 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26e428fe-2b75-4d5b-bac0-2a9dd566ff32" containerName="otc-container" Oct 31 00:24:25.572976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.572873 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="26e428fe-2b75-4d5b-bac0-2a9dd566ff32" containerName="otc-container" Oct 31 00:24:25.575797 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.575760 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" Oct 31 00:24:25.583541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.583521 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f"] Oct 31 00:24:25.587340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.587322 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" Oct 31 00:24:25.860346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:25.860274 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f"] Oct 31 00:24:25.863270 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:24:25.863242 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97c58f06_681f_4a0a_8c92_969c42255072.slice/crio-005e66aa6ac64829265c7efe6c71ead54f99c2c9d63b95022993070a4237c271 WatchSource:0}: Error finding container 005e66aa6ac64829265c7efe6c71ead54f99c2c9d63b95022993070a4237c271: Status 404 returned error can't find the container with id 005e66aa6ac64829265c7efe6c71ead54f99c2c9d63b95022993070a4237c271 Oct 31 00:24:26.514116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:26.514079 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" event={"ID":"97c58f06-681f-4a0a-8c92-969c42255072","Type":"ContainerStarted","Data":"8b5b08bebce0bd2535d806a60f589d14fa69c34f8998e759e2c5b685a35bc3c1"} Oct 31 00:24:26.514305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:26.514126 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" event={"ID":"97c58f06-681f-4a0a-8c92-969c42255072","Type":"ContainerStarted","Data":"005e66aa6ac64829265c7efe6c71ead54f99c2c9d63b95022993070a4237c271"} Oct 31 00:24:26.530599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:26.530556 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" podStartSLOduration=1.530543432 podStartE2EDuration="1.530543432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:26.529723657 +0000 UTC m=+1550.590498353" watchObservedRunningTime="2026-10-31 00:24:26.530543432 +0000 UTC m=+1550.591318126" Oct 31 00:24:32.857560 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:32.857535 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:24:33.005876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.005842 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca\" (UniqueName: \"kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca\") pod \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " Oct 31 00:24:33.006035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.005905 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv\" (UniqueName: \"kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv\") pod \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\" (UID: \"4d04f4b8-37a7-4d91-8806-ad361b32d1c0\") " Oct 31 00:24:33.006179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.006156 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca" pod "4d04f4b8-37a7-4d91-8806-ad361b32d1c0" (UID: "4d04f4b8-37a7-4d91-8806-ad361b32d1c0"). InnerVolumeSpecName "metrics-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:33.008122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.008102 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv" pod "4d04f4b8-37a7-4d91-8806-ad361b32d1c0" (UID: "4d04f4b8-37a7-4d91-8806-ad361b32d1c0"). InnerVolumeSpecName "kube-api-access-wpskv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:33.107176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.107154 5073 reconciler_common.go:299] "Volume detached for volume \"metrics-ca\" (UniqueName: \"kubernetes.io/configmap/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-metrics-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:33.107265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.107176 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpskv\" (UniqueName: \"kubernetes.io/projected/4d04f4b8-37a7-4d91-8806-ad361b32d1c0-kube-api-access-wpskv\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:33.547246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.547217 5073 generic.go:356] "Generic (PLEG): container finished" podID="4d04f4b8-37a7-4d91-8806-ad361b32d1c0" containerID="689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a" exitCode=137 Oct 31 00:24:33.547417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.547264 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/operator-metrics-auth-probe" Oct 31 00:24:33.547417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.547285 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"4d04f4b8-37a7-4d91-8806-ad361b32d1c0","Type":"ContainerDied","Data":"689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a"} Oct 31 00:24:33.547417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.547315 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/operator-metrics-auth-probe" event={"ID":"4d04f4b8-37a7-4d91-8806-ad361b32d1c0","Type":"ContainerDied","Data":"10c00d000875957c531645e01877ad06c8ab773b328a0bff649bd2da145592b3"} Oct 31 00:24:33.547417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.547331 5073 scope.go:122] "RemoveContainer" containerID="689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a" Oct 31 00:24:33.556571 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.556552 5073 scope.go:122] "RemoveContainer" containerID="689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a" Oct 31 00:24:33.556846 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:24:33.556826 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a\": container with ID starting with 689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a not found: ID does not exist" containerID="689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a" Oct 31 00:24:33.556884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.556855 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a"} err="failed to get container status \"689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a\": rpc error: code = NotFound desc = could not find container \"689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a\": container with ID starting with 689a2d658c529be73deeefe5e8fa049ca84b914d4c2ae4e6a08b20e4caf4220a not found: ID does not exist" Oct 31 00:24:33.569158 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.569137 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 31 00:24:33.573698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:33.573676 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/operator-metrics-auth-probe"] Oct 31 00:24:34.058544 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:34.058517 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d04f4b8-37a7-4d91-8806-ad361b32d1c0" path="/var/lib/kubelet/pods/4d04f4b8-37a7-4d91-8806-ad361b32d1c0/volumes" Oct 31 00:24:36.562044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:36.562016 5073 generic.go:356] "Generic (PLEG): container finished" podID="97c58f06-681f-4a0a-8c92-969c42255072" containerID="8b5b08bebce0bd2535d806a60f589d14fa69c34f8998e759e2c5b685a35bc3c1" exitCode=0 Oct 31 00:24:36.562348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:36.562090 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" event={"ID":"97c58f06-681f-4a0a-8c92-969c42255072","Type":"ContainerDied","Data":"8b5b08bebce0bd2535d806a60f589d14fa69c34f8998e759e2c5b685a35bc3c1"} Oct 31 00:24:37.696981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:37.696957 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" Oct 31 00:24:38.572586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:38.572550 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" event={"ID":"97c58f06-681f-4a0a-8c92-969c42255072","Type":"ContainerDied","Data":"005e66aa6ac64829265c7efe6c71ead54f99c2c9d63b95022993070a4237c271"} Oct 31 00:24:38.572586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:38.572573 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f" Oct 31 00:24:38.572810 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:38.572596 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005e66aa6ac64829265c7efe6c71ead54f99c2c9d63b95022993070a4237c271" Oct 31 00:24:39.907838 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.907802 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba"] Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.908740 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4d04f4b8-37a7-4d91-8806-ad361b32d1c0" containerName="curl" Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.908812 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d04f4b8-37a7-4d91-8806-ad361b32d1c0" containerName="curl" Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.909000 5073 cpu_manager.go:414] "containerMap: removing container" podUID="97c58f06-681f-4a0a-8c92-969c42255072" containerName="curl" Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.909009 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c58f06-681f-4a0a-8c92-969c42255072" containerName="curl" Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.909332 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c58f06-681f-4a0a-8c92-969c42255072" containerName="curl" Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.909346 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d04f4b8-37a7-4d91-8806-ad361b32d1c0" containerName="curl" Oct 31 00:24:39.914675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.913438 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" Oct 31 00:24:39.919544 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.919508 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba"] Oct 31 00:24:39.926683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:39.926659 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" Oct 31 00:24:40.201130 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:40.201055 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba"] Oct 31 00:24:40.203944 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:24:40.203918 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode21e56f1_66b2_4888_8b1f_8644287b71f7.slice/crio-b93ba7cb1f88368f4feac36748d1f024fd6c4f2d84e87f6f648a703e341ab478 WatchSource:0}: Error finding container b93ba7cb1f88368f4feac36748d1f024fd6c4f2d84e87f6f648a703e341ab478: Status 404 returned error can't find the container with id b93ba7cb1f88368f4feac36748d1f024fd6c4f2d84e87f6f648a703e341ab478 Oct 31 00:24:40.583982 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:40.583953 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" event={"ID":"e21e56f1-66b2-4888-8b1f-8644287b71f7","Type":"ContainerStarted","Data":"8e8122892897d14ecd40f5f84e36ddc31397d41a1f1dab81a30508328c01784d"} Oct 31 00:24:40.584149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:40.583993 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" event={"ID":"e21e56f1-66b2-4888-8b1f-8644287b71f7","Type":"ContainerStarted","Data":"b93ba7cb1f88368f4feac36748d1f024fd6c4f2d84e87f6f648a703e341ab478"} Oct 31 00:24:40.601420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:40.601361 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" podStartSLOduration=1.601345275 podStartE2EDuration="1.601345275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:40.600688691 +0000 UTC m=+1564.661463388" watchObservedRunningTime="2026-10-31 00:24:40.601345275 +0000 UTC m=+1564.662119971" Oct 31 00:24:43.085930 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:24:43.085893 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:24:45.067533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.067497 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg"] Oct 31 00:24:45.071125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.071105 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.080425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.080400 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg"] Oct 31 00:24:45.200158 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.200132 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.200289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.200165 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.200289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.200229 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlckw\" (UniqueName: \"kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.300855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.300829 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.300968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.300863 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.300968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.300917 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlckw\" (UniqueName: \"kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.301526 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.301474 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.303383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.303362 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.313647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.313621 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlckw\" (UniqueName: \"kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw\") pod \"data-science-collector-targetallocator-55b78f4f4d-qlzsg\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.382988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.382921 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:45.661159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.661066 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg"] Oct 31 00:24:45.664642 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:24:45.664441 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf81481e_7e3c_45f8_ad60_0129f1a6bcd6.slice/crio-28d0155ef0768fe1700b9b90d3474abc72177eb5d32f90a436c142b3d84c97cf WatchSource:0}: Error finding container 28d0155ef0768fe1700b9b90d3474abc72177eb5d32f90a436c142b3d84c97cf: Status 404 returned error can't find the container with id 28d0155ef0768fe1700b9b90d3474abc72177eb5d32f90a436c142b3d84c97cf Oct 31 00:24:45.672989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.672947 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:24:45.673092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:45.673022 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:24:46.610215 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.610179 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" event={"ID":"af81481e-7e3c-45f8-ad60-0129f1a6bcd6","Type":"ContainerStarted","Data":"75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59"} Oct 31 00:24:46.610674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.610227 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" event={"ID":"af81481e-7e3c-45f8-ad60-0129f1a6bcd6","Type":"ContainerStarted","Data":"28d0155ef0768fe1700b9b90d3474abc72177eb5d32f90a436c142b3d84c97cf"} Oct 31 00:24:46.610674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.610357 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:46.612133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.612112 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:24:46.631047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.631005 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" podStartSLOduration=1.630994266 podStartE2EDuration="1.630994266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:46.62960938 +0000 UTC m=+1570.690384074" watchObservedRunningTime="2026-10-31 00:24:46.630994266 +0000 UTC m=+1570.691768960" Oct 31 00:24:46.680383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.680354 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb"] Oct 31 00:24:46.680568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.680531 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" podUID="4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" containerName="ta-container" containerID="cri-o://88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07" gracePeriod=30 Oct 31 00:24:46.924373 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:46.924353 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:24:47.016298 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.016271 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx\" (UniqueName: \"kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx\") pod \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " Oct 31 00:24:47.016441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.016311 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert\") pod \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " Oct 31 00:24:47.016441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.016357 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal\" (UniqueName: \"kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal\") pod \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\" (UID: \"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d\") " Oct 31 00:24:47.016754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.016731 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal" pod "4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" (UID: "4e63450e-d2d6-4a4f-bae9-4c59cbd9305d"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:47.018602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.018580 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert" pod "4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" (UID: "4e63450e-d2d6-4a4f-bae9-4c59cbd9305d"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:24:47.018675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.018658 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx" pod "4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" (UID: "4e63450e-d2d6-4a4f-bae9-4c59cbd9305d"). InnerVolumeSpecName "kube-api-access-hthfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:47.117639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.117606 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hthfx\" (UniqueName: \"kubernetes.io/projected/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-kube-api-access-hthfx\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:47.117639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.117632 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:47.117639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.117643 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:47.616806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.616753 5073 generic.go:356] "Generic (PLEG): container finished" podID="4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" containerID="88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07" exitCode=0 Oct 31 00:24:47.617187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.616817 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" Oct 31 00:24:47.617187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.616823 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" event={"ID":"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d","Type":"ContainerDied","Data":"88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07"} Oct 31 00:24:47.617187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.616870 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb" event={"ID":"4e63450e-d2d6-4a4f-bae9-4c59cbd9305d","Type":"ContainerDied","Data":"97b08874a0cf3ad30714ea41f3b180ad9d3810cd2cf557b96fc12ee4cd469f6d"} Oct 31 00:24:47.617187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.616888 5073 scope.go:122] "RemoveContainer" containerID="88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07" Oct 31 00:24:47.626444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.626424 5073 scope.go:122] "RemoveContainer" containerID="88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07" Oct 31 00:24:47.626689 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:24:47.626673 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07\": container with ID starting with 88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07 not found: ID does not exist" containerID="88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07" Oct 31 00:24:47.626752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.626697 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07"} err="failed to get container status \"88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07\": rpc error: code = NotFound desc = could not find container \"88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07\": container with ID starting with 88dce43659710290d4491782ca548118cca3eeb7c1639149179f9d38f67eaf07 not found: ID does not exist" Oct 31 00:24:47.639038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.639015 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb"] Oct 31 00:24:47.644371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:47.644347 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-hccdb"] Oct 31 00:24:48.058182 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:48.058143 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" path="/var/lib/kubelet/pods/4e63450e-d2d6-4a4f-bae9-4c59cbd9305d/volumes" Oct 31 00:24:48.624297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:48.624265 5073 generic.go:356] "Generic (PLEG): container finished" podID="e21e56f1-66b2-4888-8b1f-8644287b71f7" containerID="8e8122892897d14ecd40f5f84e36ddc31397d41a1f1dab81a30508328c01784d" exitCode=0 Oct 31 00:24:48.624764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:48.624337 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" event={"ID":"e21e56f1-66b2-4888-8b1f-8644287b71f7","Type":"ContainerDied","Data":"8e8122892897d14ecd40f5f84e36ddc31397d41a1f1dab81a30508328c01784d"} Oct 31 00:24:49.757706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:49.757685 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" Oct 31 00:24:50.100110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100078 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d"] Oct 31 00:24:50.100458 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100445 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" containerName="ta-container" Oct 31 00:24:50.100506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100460 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" containerName="ta-container" Oct 31 00:24:50.100506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100480 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e21e56f1-66b2-4888-8b1f-8644287b71f7" containerName="curl" Oct 31 00:24:50.100506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100485 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21e56f1-66b2-4888-8b1f-8644287b71f7" containerName="curl" Oct 31 00:24:50.100594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100547 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e63450e-d2d6-4a4f-bae9-4c59cbd9305d" containerName="ta-container" Oct 31 00:24:50.100594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.100557 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21e56f1-66b2-4888-8b1f-8644287b71f7" containerName="curl" Oct 31 00:24:50.103429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.103408 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" Oct 31 00:24:50.114244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.114218 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" Oct 31 00:24:50.115591 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.115569 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d"] Oct 31 00:24:50.391131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.391105 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d"] Oct 31 00:24:50.393818 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:24:50.393773 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf7d79d2_fef6_44eb_95bc_e5066b2b8567.slice/crio-ed7138c822c676ba4873b0f3db02fe0601c0a865fea055b6cf56431a24705fc5 WatchSource:0}: Error finding container ed7138c822c676ba4873b0f3db02fe0601c0a865fea055b6cf56431a24705fc5: Status 404 returned error can't find the container with id ed7138c822c676ba4873b0f3db02fe0601c0a865fea055b6cf56431a24705fc5 Oct 31 00:24:50.634857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.634821 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" event={"ID":"df7d79d2-fef6-44eb-95bc-e5066b2b8567","Type":"ContainerStarted","Data":"8ed691c21556a1e3c9be4e761e42e7ec10ac49a9652d3b7f5bf489dd8a210ab2"} Oct 31 00:24:50.634857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.634872 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" event={"ID":"df7d79d2-fef6-44eb-95bc-e5066b2b8567","Type":"ContainerStarted","Data":"ed7138c822c676ba4873b0f3db02fe0601c0a865fea055b6cf56431a24705fc5"} Oct 31 00:24:50.636050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.636029 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" Oct 31 00:24:50.636159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.636031 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba" event={"ID":"e21e56f1-66b2-4888-8b1f-8644287b71f7","Type":"ContainerDied","Data":"b93ba7cb1f88368f4feac36748d1f024fd6c4f2d84e87f6f648a703e341ab478"} Oct 31 00:24:50.636159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.636148 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93ba7cb1f88368f4feac36748d1f024fd6c4f2d84e87f6f648a703e341ab478" Oct 31 00:24:50.651822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:50.651750 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" podStartSLOduration=0.651737701 podStartE2EDuration="651.737701ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:50.650726135 +0000 UTC m=+1574.711500831" watchObservedRunningTime="2026-10-31 00:24:50.651737701 +0000 UTC m=+1574.712512454" Oct 31 00:24:58.673328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:58.673297 5073 generic.go:356] "Generic (PLEG): container finished" podID="df7d79d2-fef6-44eb-95bc-e5066b2b8567" containerID="8ed691c21556a1e3c9be4e761e42e7ec10ac49a9652d3b7f5bf489dd8a210ab2" exitCode=0 Oct 31 00:24:58.673752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:58.673353 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" event={"ID":"df7d79d2-fef6-44eb-95bc-e5066b2b8567","Type":"ContainerDied","Data":"8ed691c21556a1e3c9be4e761e42e7ec10ac49a9652d3b7f5bf489dd8a210ab2"} Oct 31 00:24:59.816331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:24:59.816305 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" Oct 31 00:25:00.291626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.291591 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e"] Oct 31 00:25:00.291953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.291940 5073 cpu_manager.go:414] "containerMap: removing container" podUID="df7d79d2-fef6-44eb-95bc-e5066b2b8567" containerName="curl" Oct 31 00:25:00.292004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.291954 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7d79d2-fef6-44eb-95bc-e5066b2b8567" containerName="curl" Oct 31 00:25:00.292044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.292029 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7d79d2-fef6-44eb-95bc-e5066b2b8567" containerName="curl" Oct 31 00:25:00.295016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.295001 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" Oct 31 00:25:00.302387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.302358 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e"] Oct 31 00:25:00.306933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.306916 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" Oct 31 00:25:00.581158 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.581054 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e"] Oct 31 00:25:00.583827 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:25:00.583775 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65e96b98_2e5e_4af4_acf9_87bfd9f221bb.slice/crio-2da020bd0119a959a42e1cdb6f2fe76f35f5e13f1ef59fe32860e158d8f8603a WatchSource:0}: Error finding container 2da020bd0119a959a42e1cdb6f2fe76f35f5e13f1ef59fe32860e158d8f8603a: Status 404 returned error can't find the container with id 2da020bd0119a959a42e1cdb6f2fe76f35f5e13f1ef59fe32860e158d8f8603a Oct 31 00:25:00.683479 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.683459 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" Oct 31 00:25:00.683606 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.683483 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d" event={"ID":"df7d79d2-fef6-44eb-95bc-e5066b2b8567","Type":"ContainerDied","Data":"ed7138c822c676ba4873b0f3db02fe0601c0a865fea055b6cf56431a24705fc5"} Oct 31 00:25:00.683606 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.683522 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7138c822c676ba4873b0f3db02fe0601c0a865fea055b6cf56431a24705fc5" Oct 31 00:25:00.685140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.685117 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" event={"ID":"65e96b98-2e5e-4af4-acf9-87bfd9f221bb","Type":"ContainerStarted","Data":"cf856148894c62c19c5cdcd31371df47e3723d62320700eba80d376667d35a61"} Oct 31 00:25:00.685307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.685279 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" event={"ID":"65e96b98-2e5e-4af4-acf9-87bfd9f221bb","Type":"ContainerStarted","Data":"2da020bd0119a959a42e1cdb6f2fe76f35f5e13f1ef59fe32860e158d8f8603a"} Oct 31 00:25:00.702291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:00.702246 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" podStartSLOduration=0.702227107 podStartE2EDuration="702.227107ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:00.701404904 +0000 UTC m=+1584.762179601" watchObservedRunningTime="2026-10-31 00:25:00.702227107 +0000 UTC m=+1584.763001798" Oct 31 00:25:08.727342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:08.727302 5073 generic.go:356] "Generic (PLEG): container finished" podID="65e96b98-2e5e-4af4-acf9-87bfd9f221bb" containerID="cf856148894c62c19c5cdcd31371df47e3723d62320700eba80d376667d35a61" exitCode=0 Oct 31 00:25:08.727742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:08.727372 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" event={"ID":"65e96b98-2e5e-4af4-acf9-87bfd9f221bb","Type":"ContainerDied","Data":"cf856148894c62c19c5cdcd31371df47e3723d62320700eba80d376667d35a61"} Oct 31 00:25:09.863110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:09.863085 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" Oct 31 00:25:10.485002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.484973 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903"] Oct 31 00:25:10.485343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.485330 5073 cpu_manager.go:414] "containerMap: removing container" podUID="65e96b98-2e5e-4af4-acf9-87bfd9f221bb" containerName="curl" Oct 31 00:25:10.485382 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.485345 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e96b98-2e5e-4af4-acf9-87bfd9f221bb" containerName="curl" Oct 31 00:25:10.485432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.485415 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e96b98-2e5e-4af4-acf9-87bfd9f221bb" containerName="curl" Oct 31 00:25:10.488515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.488497 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" Oct 31 00:25:10.496210 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.496190 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903"] Oct 31 00:25:10.500221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.500203 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" Oct 31 00:25:10.737583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.737502 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" event={"ID":"65e96b98-2e5e-4af4-acf9-87bfd9f221bb","Type":"ContainerDied","Data":"2da020bd0119a959a42e1cdb6f2fe76f35f5e13f1ef59fe32860e158d8f8603a"} Oct 31 00:25:10.737583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.737544 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e" Oct 31 00:25:10.737764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.737546 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da020bd0119a959a42e1cdb6f2fe76f35f5e13f1ef59fe32860e158d8f8603a" Oct 31 00:25:10.777143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:10.776834 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903"] Oct 31 00:25:10.779498 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:25:10.779475 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd422f5ff_1bce_44ed_8118_0e3c007891ab.slice/crio-25980cfdbeae0d4456817534a021916d4483e83e0fc8d8378137d0cf6491a115 WatchSource:0}: Error finding container 25980cfdbeae0d4456817534a021916d4483e83e0fc8d8378137d0cf6491a115: Status 404 returned error can't find the container with id 25980cfdbeae0d4456817534a021916d4483e83e0fc8d8378137d0cf6491a115 Oct 31 00:25:11.744064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:11.744025 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" event={"ID":"d422f5ff-1bce-44ed-8118-0e3c007891ab","Type":"ContainerStarted","Data":"04198ff38e965e80dde161d80107c0531064422d765b8ba73236f4edb20855da"} Oct 31 00:25:11.744064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:11.744070 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" event={"ID":"d422f5ff-1bce-44ed-8118-0e3c007891ab","Type":"ContainerStarted","Data":"25980cfdbeae0d4456817534a021916d4483e83e0fc8d8378137d0cf6491a115"} Oct 31 00:25:11.765439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:11.765388 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" podStartSLOduration=1.765374266 podStartE2EDuration="1.765374266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:11.764855007 +0000 UTC m=+1595.825629703" watchObservedRunningTime="2026-10-31 00:25:11.765374266 +0000 UTC m=+1595.826148960" Oct 31 00:25:13.085243 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:25:13.085210 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:25:18.778636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:18.778603 5073 generic.go:356] "Generic (PLEG): container finished" podID="d422f5ff-1bce-44ed-8118-0e3c007891ab" containerID="04198ff38e965e80dde161d80107c0531064422d765b8ba73236f4edb20855da" exitCode=0 Oct 31 00:25:18.779053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:18.778670 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" event={"ID":"d422f5ff-1bce-44ed-8118-0e3c007891ab","Type":"ContainerDied","Data":"04198ff38e965e80dde161d80107c0531064422d765b8ba73236f4edb20855da"} Oct 31 00:25:19.916436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:19.916413 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" Oct 31 00:25:20.677621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.677589 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa"] Oct 31 00:25:20.678044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.678026 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d422f5ff-1bce-44ed-8118-0e3c007891ab" containerName="curl" Oct 31 00:25:20.678138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.678046 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d422f5ff-1bce-44ed-8118-0e3c007891ab" containerName="curl" Oct 31 00:25:20.678194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.678147 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d422f5ff-1bce-44ed-8118-0e3c007891ab" containerName="curl" Oct 31 00:25:20.681235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.681216 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" Oct 31 00:25:20.688206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.688187 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa"] Oct 31 00:25:20.693195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.693176 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" Oct 31 00:25:20.791047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.791009 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" event={"ID":"d422f5ff-1bce-44ed-8118-0e3c007891ab","Type":"ContainerDied","Data":"25980cfdbeae0d4456817534a021916d4483e83e0fc8d8378137d0cf6491a115"} Oct 31 00:25:20.791047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.791050 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25980cfdbeae0d4456817534a021916d4483e83e0fc8d8378137d0cf6491a115" Oct 31 00:25:20.791271 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.791102 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903" Oct 31 00:25:20.972004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:20.971940 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa"] Oct 31 00:25:20.975084 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:25:20.975057 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode093c8aa_677e_4518_af69_39ef54e79ab1.slice/crio-c51ba264dabef4ae41021724a493bd79a2169235d45b8cb4ac6a06c4cd8fd013 WatchSource:0}: Error finding container c51ba264dabef4ae41021724a493bd79a2169235d45b8cb4ac6a06c4cd8fd013: Status 404 returned error can't find the container with id c51ba264dabef4ae41021724a493bd79a2169235d45b8cb4ac6a06c4cd8fd013 Oct 31 00:25:21.796901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:21.796869 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" event={"ID":"e093c8aa-677e-4518-af69-39ef54e79ab1","Type":"ContainerStarted","Data":"3aa0264bd961e1e3dec3c22b0acfadbc330fe53dd8799129711c2650c0825d3f"} Oct 31 00:25:21.797085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:21.796916 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" event={"ID":"e093c8aa-677e-4518-af69-39ef54e79ab1","Type":"ContainerStarted","Data":"c51ba264dabef4ae41021724a493bd79a2169235d45b8cb4ac6a06c4cd8fd013"} Oct 31 00:25:21.814790 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:21.814746 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" podStartSLOduration=1.8147316679999999 podStartE2EDuration="1.814731668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:21.814110833 +0000 UTC m=+1605.874885529" watchObservedRunningTime="2026-10-31 00:25:21.814731668 +0000 UTC m=+1605.875506363" Oct 31 00:25:28.834638 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:28.834563 5073 generic.go:356] "Generic (PLEG): container finished" podID="e093c8aa-677e-4518-af69-39ef54e79ab1" containerID="3aa0264bd961e1e3dec3c22b0acfadbc330fe53dd8799129711c2650c0825d3f" exitCode=0 Oct 31 00:25:28.834638 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:28.834614 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" event={"ID":"e093c8aa-677e-4518-af69-39ef54e79ab1","Type":"ContainerDied","Data":"3aa0264bd961e1e3dec3c22b0acfadbc330fe53dd8799129711c2650c0825d3f"} Oct 31 00:25:29.968983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:29.968960 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" Oct 31 00:25:30.844815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.844767 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" event={"ID":"e093c8aa-677e-4518-af69-39ef54e79ab1","Type":"ContainerDied","Data":"c51ba264dabef4ae41021724a493bd79a2169235d45b8cb4ac6a06c4cd8fd013"} Oct 31 00:25:30.844983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.844824 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51ba264dabef4ae41021724a493bd79a2169235d45b8cb4ac6a06c4cd8fd013" Oct 31 00:25:30.844983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.844776 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa" Oct 31 00:25:30.876747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.876719 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64"] Oct 31 00:25:30.877085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.877072 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e093c8aa-677e-4518-af69-39ef54e79ab1" containerName="curl" Oct 31 00:25:30.877124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.877086 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e093c8aa-677e-4518-af69-39ef54e79ab1" containerName="curl" Oct 31 00:25:30.877162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.877143 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e093c8aa-677e-4518-af69-39ef54e79ab1" containerName="curl" Oct 31 00:25:30.880020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.880006 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" Oct 31 00:25:30.887819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.887777 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64"] Oct 31 00:25:30.891287 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:30.891269 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" Oct 31 00:25:31.170040 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:31.169968 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64"] Oct 31 00:25:31.172719 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:25:31.172694 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6086f36_26b5_4cd1_8a0f_3abbeec070ef.slice/crio-49927543d62f6a436dafe7834ae64f4a1036b21944ebd26cc40fc3ac0a3e66a1 WatchSource:0}: Error finding container 49927543d62f6a436dafe7834ae64f4a1036b21944ebd26cc40fc3ac0a3e66a1: Status 404 returned error can't find the container with id 49927543d62f6a436dafe7834ae64f4a1036b21944ebd26cc40fc3ac0a3e66a1 Oct 31 00:25:31.851177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:31.851143 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" event={"ID":"a6086f36-26b5-4cd1-8a0f-3abbeec070ef","Type":"ContainerStarted","Data":"7d4c0c198aefc4d0722278c1030bc3a178a914ac0707e2f62c1de4c4b49bc456"} Oct 31 00:25:31.851177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:31.851187 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" event={"ID":"a6086f36-26b5-4cd1-8a0f-3abbeec070ef","Type":"ContainerStarted","Data":"49927543d62f6a436dafe7834ae64f4a1036b21944ebd26cc40fc3ac0a3e66a1"} Oct 31 00:25:31.868480 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:31.868434 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" podStartSLOduration=1.868418751 podStartE2EDuration="1.868418751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:31.867125806 +0000 UTC m=+1615.927900514" watchObservedRunningTime="2026-10-31 00:25:31.868418751 +0000 UTC m=+1615.929193446" Oct 31 00:25:38.884925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:38.884891 5073 generic.go:356] "Generic (PLEG): container finished" podID="a6086f36-26b5-4cd1-8a0f-3abbeec070ef" containerID="7d4c0c198aefc4d0722278c1030bc3a178a914ac0707e2f62c1de4c4b49bc456" exitCode=0 Oct 31 00:25:38.885234 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:38.884961 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" event={"ID":"a6086f36-26b5-4cd1-8a0f-3abbeec070ef","Type":"ContainerDied","Data":"7d4c0c198aefc4d0722278c1030bc3a178a914ac0707e2f62c1de4c4b49bc456"} Oct 31 00:25:40.026324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:40.026302 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" Oct 31 00:25:40.895430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:40.895403 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" Oct 31 00:25:40.895430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:40.895415 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64" event={"ID":"a6086f36-26b5-4cd1-8a0f-3abbeec070ef","Type":"ContainerDied","Data":"49927543d62f6a436dafe7834ae64f4a1036b21944ebd26cc40fc3ac0a3e66a1"} Oct 31 00:25:40.895709 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:40.895447 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49927543d62f6a436dafe7834ae64f4a1036b21944ebd26cc40fc3ac0a3e66a1" Oct 31 00:25:41.072053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.072017 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6"] Oct 31 00:25:41.072547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.072503 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a6086f36-26b5-4cd1-8a0f-3abbeec070ef" containerName="curl" Oct 31 00:25:41.072547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.072523 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6086f36-26b5-4cd1-8a0f-3abbeec070ef" containerName="curl" Oct 31 00:25:41.072681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.072666 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6086f36-26b5-4cd1-8a0f-3abbeec070ef" containerName="curl" Oct 31 00:25:41.075619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.075599 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" Oct 31 00:25:41.083099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.083077 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6"] Oct 31 00:25:41.087259 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.087240 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" Oct 31 00:25:41.363272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.363211 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6"] Oct 31 00:25:41.366044 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:25:41.366003 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67187bc6_e774_4aba_a8d0_49dd42a91fa4.slice/crio-909104f5ed1fc0b7eec1416d078b02195cf80747f746031ce96768156db21b45 WatchSource:0}: Error finding container 909104f5ed1fc0b7eec1416d078b02195cf80747f746031ce96768156db21b45: Status 404 returned error can't find the container with id 909104f5ed1fc0b7eec1416d078b02195cf80747f746031ce96768156db21b45 Oct 31 00:25:41.901139 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.901108 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" event={"ID":"67187bc6-e774-4aba-a8d0-49dd42a91fa4","Type":"ContainerStarted","Data":"917a92164e28d09bcd81c6cc33169554933c7a298dc978a2d262ff9a1f7e3189"} Oct 31 00:25:41.901286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.901157 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" event={"ID":"67187bc6-e774-4aba-a8d0-49dd42a91fa4","Type":"ContainerStarted","Data":"909104f5ed1fc0b7eec1416d078b02195cf80747f746031ce96768156db21b45"} Oct 31 00:25:41.917500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:41.917454 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" podStartSLOduration=0.917438633 podStartE2EDuration="917.438633ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:41.917173888 +0000 UTC m=+1625.977948584" watchObservedRunningTime="2026-10-31 00:25:41.917438633 +0000 UTC m=+1625.978213329" Oct 31 00:25:43.084238 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:25:43.084196 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:25:49.941143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:49.941103 5073 generic.go:356] "Generic (PLEG): container finished" podID="67187bc6-e774-4aba-a8d0-49dd42a91fa4" containerID="917a92164e28d09bcd81c6cc33169554933c7a298dc978a2d262ff9a1f7e3189" exitCode=0 Oct 31 00:25:49.941625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:49.941181 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" event={"ID":"67187bc6-e774-4aba-a8d0-49dd42a91fa4","Type":"ContainerDied","Data":"917a92164e28d09bcd81c6cc33169554933c7a298dc978a2d262ff9a1f7e3189"} Oct 31 00:25:51.076193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.076164 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" Oct 31 00:25:51.266576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.266543 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1"] Oct 31 00:25:51.266983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.266969 5073 cpu_manager.go:414] "containerMap: removing container" podUID="67187bc6-e774-4aba-a8d0-49dd42a91fa4" containerName="curl" Oct 31 00:25:51.267046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.266986 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67187bc6-e774-4aba-a8d0-49dd42a91fa4" containerName="curl" Oct 31 00:25:51.267088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.267055 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="67187bc6-e774-4aba-a8d0-49dd42a91fa4" containerName="curl" Oct 31 00:25:51.269941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.269927 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" Oct 31 00:25:51.277248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.277223 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1"] Oct 31 00:25:51.281753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.281732 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" Oct 31 00:25:51.557346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.557275 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1"] Oct 31 00:25:51.563601 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:25:51.560475 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ea59b23_25b1_40bb_b02b_d8993605f0c5.slice/crio-ca9a05d1c1eb903193074dd68cc0afc62c6f57b025c5eeb6ff4181f59f6f024b WatchSource:0}: Error finding container ca9a05d1c1eb903193074dd68cc0afc62c6f57b025c5eeb6ff4181f59f6f024b: Status 404 returned error can't find the container with id ca9a05d1c1eb903193074dd68cc0afc62c6f57b025c5eeb6ff4181f59f6f024b Oct 31 00:25:51.951877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.951775 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" Oct 31 00:25:51.951877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.951815 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6" event={"ID":"67187bc6-e774-4aba-a8d0-49dd42a91fa4","Type":"ContainerDied","Data":"909104f5ed1fc0b7eec1416d078b02195cf80747f746031ce96768156db21b45"} Oct 31 00:25:51.951877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.951857 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909104f5ed1fc0b7eec1416d078b02195cf80747f746031ce96768156db21b45" Oct 31 00:25:51.953279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.953258 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" event={"ID":"6ea59b23-25b1-40bb-b02b-d8993605f0c5","Type":"ContainerStarted","Data":"8caa146a1c6524547c3e0cb5ae2d34d0763225d08dce8c3446efd2da9b7768d6"} Oct 31 00:25:51.953279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.953287 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" event={"ID":"6ea59b23-25b1-40bb-b02b-d8993605f0c5","Type":"ContainerStarted","Data":"ca9a05d1c1eb903193074dd68cc0afc62c6f57b025c5eeb6ff4181f59f6f024b"} Oct 31 00:25:51.970768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:51.970729 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" podStartSLOduration=0.970718016 podStartE2EDuration="970.718016ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:51.97010512 +0000 UTC m=+1636.030879815" watchObservedRunningTime="2026-10-31 00:25:51.970718016 +0000 UTC m=+1636.031492729" Oct 31 00:25:59.990576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:59.990541 5073 generic.go:356] "Generic (PLEG): container finished" podID="6ea59b23-25b1-40bb-b02b-d8993605f0c5" containerID="8caa146a1c6524547c3e0cb5ae2d34d0763225d08dce8c3446efd2da9b7768d6" exitCode=0 Oct 31 00:25:59.990986 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:25:59.990617 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" event={"ID":"6ea59b23-25b1-40bb-b02b-d8993605f0c5","Type":"ContainerDied","Data":"8caa146a1c6524547c3e0cb5ae2d34d0763225d08dce8c3446efd2da9b7768d6"} Oct 31 00:26:01.130119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.130095 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" Oct 31 00:26:01.459203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.459127 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98"] Oct 31 00:26:01.459468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.459456 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6ea59b23-25b1-40bb-b02b-d8993605f0c5" containerName="curl" Oct 31 00:26:01.459509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.459470 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea59b23-25b1-40bb-b02b-d8993605f0c5" containerName="curl" Oct 31 00:26:01.459548 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.459538 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea59b23-25b1-40bb-b02b-d8993605f0c5" containerName="curl" Oct 31 00:26:01.462390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.462374 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" Oct 31 00:26:01.470313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.470288 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98"] Oct 31 00:26:01.473730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.473711 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" Oct 31 00:26:01.748060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:01.747424 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98"] Oct 31 00:26:01.753886 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:01.753860 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d7df63_e32e_4955_964e_2ed4aa29cc2a.slice/crio-bf77507070dce42f7603f37f6b1bf6bff8a1280b7982dbc2f25628e5919e94f7 WatchSource:0}: Error finding container bf77507070dce42f7603f37f6b1bf6bff8a1280b7982dbc2f25628e5919e94f7: Status 404 returned error can't find the container with id bf77507070dce42f7603f37f6b1bf6bff8a1280b7982dbc2f25628e5919e94f7 Oct 31 00:26:02.002212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:02.002130 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" event={"ID":"18d7df63-e32e-4955-964e-2ed4aa29cc2a","Type":"ContainerStarted","Data":"003e94064fa90f9abae1ea3e4a7a8aa2b9217fb22ba5ca8a0e742303645c87f9"} Oct 31 00:26:02.002212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:02.002180 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" event={"ID":"18d7df63-e32e-4955-964e-2ed4aa29cc2a","Type":"ContainerStarted","Data":"bf77507070dce42f7603f37f6b1bf6bff8a1280b7982dbc2f25628e5919e94f7"} Oct 31 00:26:02.003567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:02.003541 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" event={"ID":"6ea59b23-25b1-40bb-b02b-d8993605f0c5","Type":"ContainerDied","Data":"ca9a05d1c1eb903193074dd68cc0afc62c6f57b025c5eeb6ff4181f59f6f024b"} Oct 31 00:26:02.003675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:02.003575 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca9a05d1c1eb903193074dd68cc0afc62c6f57b025c5eeb6ff4181f59f6f024b" Oct 31 00:26:02.003675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:02.003555 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1" Oct 31 00:26:02.019124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:02.019086 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" podStartSLOduration=1.019039664 podStartE2EDuration="1.019039664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:02.017549736 +0000 UTC m=+1646.078324432" watchObservedRunningTime="2026-10-31 00:26:02.019039664 +0000 UTC m=+1646.079814402" Oct 31 00:26:10.042754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:10.042715 5073 generic.go:356] "Generic (PLEG): container finished" podID="18d7df63-e32e-4955-964e-2ed4aa29cc2a" containerID="003e94064fa90f9abae1ea3e4a7a8aa2b9217fb22ba5ca8a0e742303645c87f9" exitCode=0 Oct 31 00:26:10.043186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:10.042811 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" event={"ID":"18d7df63-e32e-4955-964e-2ed4aa29cc2a","Type":"ContainerDied","Data":"003e94064fa90f9abae1ea3e4a7a8aa2b9217fb22ba5ca8a0e742303645c87f9"} Oct 31 00:26:11.179618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.179597 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" Oct 31 00:26:11.653045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.653015 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658"] Oct 31 00:26:11.653407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.653394 5073 cpu_manager.go:414] "containerMap: removing container" podUID="18d7df63-e32e-4955-964e-2ed4aa29cc2a" containerName="curl" Oct 31 00:26:11.653461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.653409 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d7df63-e32e-4955-964e-2ed4aa29cc2a" containerName="curl" Oct 31 00:26:11.653501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.653471 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d7df63-e32e-4955-964e-2ed4aa29cc2a" containerName="curl" Oct 31 00:26:11.656398 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.656382 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" Oct 31 00:26:11.665140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.665114 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658"] Oct 31 00:26:11.667992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.667969 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" Oct 31 00:26:11.945046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:11.945016 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658"] Oct 31 00:26:11.947756 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:11.947725 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b4e7b4f_86e7_44d5_a1ca_d605bebe2d29.slice/crio-2f6b5d18ed67ac420e2ab37e554a6b9bb1d1d7e236461f6ca6bc6bdf6780c3cd WatchSource:0}: Error finding container 2f6b5d18ed67ac420e2ab37e554a6b9bb1d1d7e236461f6ca6bc6bdf6780c3cd: Status 404 returned error can't find the container with id 2f6b5d18ed67ac420e2ab37e554a6b9bb1d1d7e236461f6ca6bc6bdf6780c3cd Oct 31 00:26:12.053230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:12.053209 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" Oct 31 00:26:12.058136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:12.058112 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98" event={"ID":"18d7df63-e32e-4955-964e-2ed4aa29cc2a","Type":"ContainerDied","Data":"bf77507070dce42f7603f37f6b1bf6bff8a1280b7982dbc2f25628e5919e94f7"} Oct 31 00:26:12.058273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:12.058144 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf77507070dce42f7603f37f6b1bf6bff8a1280b7982dbc2f25628e5919e94f7" Oct 31 00:26:12.058273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:12.058153 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" event={"ID":"8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29","Type":"ContainerStarted","Data":"c627fada6b06bafcff74aab058980e55824a0880eb32055e17a28b8190d09489"} Oct 31 00:26:12.058273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:12.058165 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" event={"ID":"8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29","Type":"ContainerStarted","Data":"2f6b5d18ed67ac420e2ab37e554a6b9bb1d1d7e236461f6ca6bc6bdf6780c3cd"} Oct 31 00:26:12.071449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:12.071403 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" podStartSLOduration=1.071385865 podStartE2EDuration="1.071385865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:12.071346545 +0000 UTC m=+1656.132121240" watchObservedRunningTime="2026-10-31 00:26:12.071385865 +0000 UTC m=+1656.132160560" Oct 31 00:26:13.085057 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:26:13.084987 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:26:20.091128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:20.091095 5073 generic.go:356] "Generic (PLEG): container finished" podID="8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29" containerID="c627fada6b06bafcff74aab058980e55824a0880eb32055e17a28b8190d09489" exitCode=0 Oct 31 00:26:20.091523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:20.091167 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" event={"ID":"8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29","Type":"ContainerDied","Data":"c627fada6b06bafcff74aab058980e55824a0880eb32055e17a28b8190d09489"} Oct 31 00:26:21.220041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.220019 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" Oct 31 00:26:21.845508 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.845476 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10"] Oct 31 00:26:21.845875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.845860 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29" containerName="curl" Oct 31 00:26:21.845875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.845875 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29" containerName="curl" Oct 31 00:26:21.845984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.845970 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29" containerName="curl" Oct 31 00:26:21.849045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.849028 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" Oct 31 00:26:21.854412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.854387 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10"] Oct 31 00:26:21.861847 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:21.861824 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" Oct 31 00:26:22.102837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:22.102766 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" Oct 31 00:26:22.102837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:22.102806 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658" event={"ID":"8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29","Type":"ContainerDied","Data":"2f6b5d18ed67ac420e2ab37e554a6b9bb1d1d7e236461f6ca6bc6bdf6780c3cd"} Oct 31 00:26:22.102996 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:22.102838 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f6b5d18ed67ac420e2ab37e554a6b9bb1d1d7e236461f6ca6bc6bdf6780c3cd" Oct 31 00:26:22.139606 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:22.139583 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10"] Oct 31 00:26:22.142517 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:22.142492 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3a65b56_e108_4eeb_8b2e_19bf3a824f65.slice/crio-7e10e31ed4975baea4fef22f2f8e974384ed5a6c699ad9645cd79b958401268f WatchSource:0}: Error finding container 7e10e31ed4975baea4fef22f2f8e974384ed5a6c699ad9645cd79b958401268f: Status 404 returned error can't find the container with id 7e10e31ed4975baea4fef22f2f8e974384ed5a6c699ad9645cd79b958401268f Oct 31 00:26:23.109711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:23.109680 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" event={"ID":"c3a65b56-e108-4eeb-8b2e-19bf3a824f65","Type":"ContainerStarted","Data":"8ac4f5d2ecb23f9dae1108dcb64925639e908269ff6e6b8e60b9f71a68cad2b3"} Oct 31 00:26:23.109711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:23.109715 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" event={"ID":"c3a65b56-e108-4eeb-8b2e-19bf3a824f65","Type":"ContainerStarted","Data":"7e10e31ed4975baea4fef22f2f8e974384ed5a6c699ad9645cd79b958401268f"} Oct 31 00:26:23.126535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:23.126475 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" podStartSLOduration=2.126460176 podStartE2EDuration="2.126460176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:23.125514565 +0000 UTC m=+1667.186289257" watchObservedRunningTime="2026-10-31 00:26:23.126460176 +0000 UTC m=+1667.187234870" Oct 31 00:26:30.142816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:30.142766 5073 generic.go:356] "Generic (PLEG): container finished" podID="c3a65b56-e108-4eeb-8b2e-19bf3a824f65" containerID="8ac4f5d2ecb23f9dae1108dcb64925639e908269ff6e6b8e60b9f71a68cad2b3" exitCode=0 Oct 31 00:26:30.143197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:30.142839 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" event={"ID":"c3a65b56-e108-4eeb-8b2e-19bf3a824f65","Type":"ContainerDied","Data":"8ac4f5d2ecb23f9dae1108dcb64925639e908269ff6e6b8e60b9f71a68cad2b3"} Oct 31 00:26:31.287066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:31.287043 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" Oct 31 00:26:32.041251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.041218 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267"] Oct 31 00:26:32.041567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.041555 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c3a65b56-e108-4eeb-8b2e-19bf3a824f65" containerName="curl" Oct 31 00:26:32.041614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.041569 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a65b56-e108-4eeb-8b2e-19bf3a824f65" containerName="curl" Oct 31 00:26:32.041666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.041658 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a65b56-e108-4eeb-8b2e-19bf3a824f65" containerName="curl" Oct 31 00:26:32.044550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.044532 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" Oct 31 00:26:32.056970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.056935 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" Oct 31 00:26:32.058685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.058664 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267"] Oct 31 00:26:32.154665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.154640 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" Oct 31 00:26:32.154812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.154639 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10" event={"ID":"c3a65b56-e108-4eeb-8b2e-19bf3a824f65","Type":"ContainerDied","Data":"7e10e31ed4975baea4fef22f2f8e974384ed5a6c699ad9645cd79b958401268f"} Oct 31 00:26:32.154812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.154749 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e10e31ed4975baea4fef22f2f8e974384ed5a6c699ad9645cd79b958401268f" Oct 31 00:26:32.330363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:32.329955 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267"] Oct 31 00:26:32.332978 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:32.332947 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7248d8fd_b22a_4dbe_8544_4adeaabf640f.slice/crio-2428cfb9c55c2502a3d1216223a9f9a7770c04a5e562bf5ef5a986d269b4bd5f WatchSource:0}: Error finding container 2428cfb9c55c2502a3d1216223a9f9a7770c04a5e562bf5ef5a986d269b4bd5f: Status 404 returned error can't find the container with id 2428cfb9c55c2502a3d1216223a9f9a7770c04a5e562bf5ef5a986d269b4bd5f Oct 31 00:26:33.160537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:33.160453 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" event={"ID":"7248d8fd-b22a-4dbe-8544-4adeaabf640f","Type":"ContainerStarted","Data":"c499cb020bf82d66bb8a869ff927edb89f7d0e09b9a0b00a0c923854c667b0a5"} Oct 31 00:26:33.160537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:33.160502 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" event={"ID":"7248d8fd-b22a-4dbe-8544-4adeaabf640f","Type":"ContainerStarted","Data":"2428cfb9c55c2502a3d1216223a9f9a7770c04a5e562bf5ef5a986d269b4bd5f"} Oct 31 00:26:33.181771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:33.181661 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" podStartSLOduration=1.181645689 podStartE2EDuration="1.181645689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:33.181279009 +0000 UTC m=+1677.242053704" watchObservedRunningTime="2026-10-31 00:26:33.181645689 +0000 UTC m=+1677.242420385" Oct 31 00:26:40.192768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:40.192693 5073 generic.go:356] "Generic (PLEG): container finished" podID="7248d8fd-b22a-4dbe-8544-4adeaabf640f" containerID="c499cb020bf82d66bb8a869ff927edb89f7d0e09b9a0b00a0c923854c667b0a5" exitCode=0 Oct 31 00:26:40.193215 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:40.192770 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" event={"ID":"7248d8fd-b22a-4dbe-8544-4adeaabf640f","Type":"ContainerDied","Data":"c499cb020bf82d66bb8a869ff927edb89f7d0e09b9a0b00a0c923854c667b0a5"} Oct 31 00:26:41.333381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:41.333351 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" Oct 31 00:26:42.204969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.204933 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" event={"ID":"7248d8fd-b22a-4dbe-8544-4adeaabf640f","Type":"ContainerDied","Data":"2428cfb9c55c2502a3d1216223a9f9a7770c04a5e562bf5ef5a986d269b4bd5f"} Oct 31 00:26:42.204969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.204953 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267" Oct 31 00:26:42.204969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.204971 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2428cfb9c55c2502a3d1216223a9f9a7770c04a5e562bf5ef5a986d269b4bd5f" Oct 31 00:26:42.239484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.239456 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550"] Oct 31 00:26:42.239814 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.239802 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7248d8fd-b22a-4dbe-8544-4adeaabf640f" containerName="curl" Oct 31 00:26:42.239861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.239816 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7248d8fd-b22a-4dbe-8544-4adeaabf640f" containerName="curl" Oct 31 00:26:42.239895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.239891 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7248d8fd-b22a-4dbe-8544-4adeaabf640f" containerName="curl" Oct 31 00:26:42.242880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.242866 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" Oct 31 00:26:42.249491 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.249468 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:26:42.251377 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.251360 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550"] Oct 31 00:26:42.254392 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.254376 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" Oct 31 00:26:42.529717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:42.529687 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550"] Oct 31 00:26:42.532751 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:42.532726 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27b8c5f4_d54c_4900_b174_ee3c499a5387.slice/crio-ea8d1c280f8a414aab3d1d51bdc38fa649abfb0648d63a324e2c7a9711b16fd9 WatchSource:0}: Error finding container ea8d1c280f8a414aab3d1d51bdc38fa649abfb0648d63a324e2c7a9711b16fd9: Status 404 returned error can't find the container with id ea8d1c280f8a414aab3d1d51bdc38fa649abfb0648d63a324e2c7a9711b16fd9 Oct 31 00:26:43.085498 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:26:43.085455 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:26:43.211445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:43.211417 5073 generic.go:356] "Generic (PLEG): container finished" podID="27b8c5f4-d54c-4900-b174-ee3c499a5387" containerID="695c265b74ffb15a37706d283246ace196342b5aa518076d9c807ae5579b4899" exitCode=0 Oct 31 00:26:43.211586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:43.211503 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" event={"ID":"27b8c5f4-d54c-4900-b174-ee3c499a5387","Type":"ContainerDied","Data":"695c265b74ffb15a37706d283246ace196342b5aa518076d9c807ae5579b4899"} Oct 31 00:26:43.211586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:43.211541 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" event={"ID":"27b8c5f4-d54c-4900-b174-ee3c499a5387","Type":"ContainerStarted","Data":"ea8d1c280f8a414aab3d1d51bdc38fa649abfb0648d63a324e2c7a9711b16fd9"} Oct 31 00:26:44.349070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:44.349045 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" Oct 31 00:26:45.223047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:45.223015 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" Oct 31 00:26:45.223247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:45.223048 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550" event={"ID":"27b8c5f4-d54c-4900-b174-ee3c499a5387","Type":"ContainerDied","Data":"ea8d1c280f8a414aab3d1d51bdc38fa649abfb0648d63a324e2c7a9711b16fd9"} Oct 31 00:26:45.223247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:45.223087 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea8d1c280f8a414aab3d1d51bdc38fa649abfb0648d63a324e2c7a9711b16fd9" Oct 31 00:26:46.380806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.380758 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa"] Oct 31 00:26:46.381221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.381111 5073 cpu_manager.go:414] "containerMap: removing container" podUID="27b8c5f4-d54c-4900-b174-ee3c499a5387" containerName="curl" Oct 31 00:26:46.381221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.381124 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b8c5f4-d54c-4900-b174-ee3c499a5387" containerName="curl" Oct 31 00:26:46.381306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.381225 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b8c5f4-d54c-4900-b174-ee3c499a5387" containerName="curl" Oct 31 00:26:46.384171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.384154 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" Oct 31 00:26:46.395366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.391331 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:26:46.395846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.395821 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa"] Oct 31 00:26:46.401444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.401422 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" Oct 31 00:26:46.675775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:46.675553 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa"] Oct 31 00:26:46.678293 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:46.678256 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828687c7_778b_4e04_8bc9_f73a509344c2.slice/crio-faec89b38f758eb2c7a866df2c4e6198d3831a6de09818f8cb6c72a3d99081d1 WatchSource:0}: Error finding container faec89b38f758eb2c7a866df2c4e6198d3831a6de09818f8cb6c72a3d99081d1: Status 404 returned error can't find the container with id faec89b38f758eb2c7a866df2c4e6198d3831a6de09818f8cb6c72a3d99081d1 Oct 31 00:26:47.235359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:47.235323 5073 generic.go:356] "Generic (PLEG): container finished" podID="828687c7-778b-4e04-8bc9-f73a509344c2" containerID="9fbdf3b21489ac283e0c825b75b27f5b53b1e979530fd1037aeb69b78aada0a6" exitCode=0 Oct 31 00:26:47.235530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:47.235404 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" event={"ID":"828687c7-778b-4e04-8bc9-f73a509344c2","Type":"ContainerDied","Data":"9fbdf3b21489ac283e0c825b75b27f5b53b1e979530fd1037aeb69b78aada0a6"} Oct 31 00:26:47.235530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:47.235445 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" event={"ID":"828687c7-778b-4e04-8bc9-f73a509344c2","Type":"ContainerStarted","Data":"faec89b38f758eb2c7a866df2c4e6198d3831a6de09818f8cb6c72a3d99081d1"} Oct 31 00:26:48.375006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.374984 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" Oct 31 00:26:48.499768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.499709 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3"] Oct 31 00:26:48.500092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.500079 5073 cpu_manager.go:414] "containerMap: removing container" podUID="828687c7-778b-4e04-8bc9-f73a509344c2" containerName="curl" Oct 31 00:26:48.500131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.500094 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828687c7-778b-4e04-8bc9-f73a509344c2" containerName="curl" Oct 31 00:26:48.500165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.500150 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="828687c7-778b-4e04-8bc9-f73a509344c2" containerName="curl" Oct 31 00:26:48.503050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.503034 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" Oct 31 00:26:48.510675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.510650 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3"] Oct 31 00:26:48.514712 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.514689 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" Oct 31 00:26:48.788762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:48.788629 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3"] Oct 31 00:26:48.791476 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:48.791451 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0469e63_7fcc_4a89_b0c9_94decba395e5.slice/crio-64a68dca1ccac66ed995aada364809a4a26ebb11eff8ad0279552fdafcf62578 WatchSource:0}: Error finding container 64a68dca1ccac66ed995aada364809a4a26ebb11eff8ad0279552fdafcf62578: Status 404 returned error can't find the container with id 64a68dca1ccac66ed995aada364809a4a26ebb11eff8ad0279552fdafcf62578 Oct 31 00:26:49.247801 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:49.247755 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" Oct 31 00:26:49.247969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:49.247772 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa" event={"ID":"828687c7-778b-4e04-8bc9-f73a509344c2","Type":"ContainerDied","Data":"faec89b38f758eb2c7a866df2c4e6198d3831a6de09818f8cb6c72a3d99081d1"} Oct 31 00:26:49.247969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:49.247836 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faec89b38f758eb2c7a866df2c4e6198d3831a6de09818f8cb6c72a3d99081d1" Oct 31 00:26:49.249345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:49.249321 5073 generic.go:356] "Generic (PLEG): container finished" podID="b0469e63-7fcc-4a89-b0c9-94decba395e5" containerID="2dd94258ad6dbc8f23dd4a0168b2a2f1216340d5aeec48ae87ac870bda400a9a" exitCode=0 Oct 31 00:26:49.249463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:49.249389 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" event={"ID":"b0469e63-7fcc-4a89-b0c9-94decba395e5","Type":"ContainerDied","Data":"2dd94258ad6dbc8f23dd4a0168b2a2f1216340d5aeec48ae87ac870bda400a9a"} Oct 31 00:26:49.249463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:49.249428 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" event={"ID":"b0469e63-7fcc-4a89-b0c9-94decba395e5","Type":"ContainerStarted","Data":"64a68dca1ccac66ed995aada364809a4a26ebb11eff8ad0279552fdafcf62578"} Oct 31 00:26:50.386159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.386133 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" Oct 31 00:26:50.608012 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.607945 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683"] Oct 31 00:26:50.608319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.608307 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b0469e63-7fcc-4a89-b0c9-94decba395e5" containerName="curl" Oct 31 00:26:50.608364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.608320 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0469e63-7fcc-4a89-b0c9-94decba395e5" containerName="curl" Oct 31 00:26:50.608403 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.608376 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0469e63-7fcc-4a89-b0c9-94decba395e5" containerName="curl" Oct 31 00:26:50.611338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.611323 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" Oct 31 00:26:50.620109 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.620087 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683"] Oct 31 00:26:50.622994 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.622975 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" Oct 31 00:26:50.898010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:50.897982 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683"] Oct 31 00:26:50.900666 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:50.900638 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2acce605_ed2a_4a6a_b4bd_82ddcec41594.slice/crio-c9e7cf9b3ebf161d9d4506ea06fa4e1689c2718ad07e6bf5752a4dc164cb291b WatchSource:0}: Error finding container c9e7cf9b3ebf161d9d4506ea06fa4e1689c2718ad07e6bf5752a4dc164cb291b: Status 404 returned error can't find the container with id c9e7cf9b3ebf161d9d4506ea06fa4e1689c2718ad07e6bf5752a4dc164cb291b Oct 31 00:26:51.261242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:51.261220 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" Oct 31 00:26:51.261393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:51.261265 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3" event={"ID":"b0469e63-7fcc-4a89-b0c9-94decba395e5","Type":"ContainerDied","Data":"64a68dca1ccac66ed995aada364809a4a26ebb11eff8ad0279552fdafcf62578"} Oct 31 00:26:51.261393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:51.261305 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a68dca1ccac66ed995aada364809a4a26ebb11eff8ad0279552fdafcf62578" Oct 31 00:26:51.262912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:51.262891 5073 generic.go:356] "Generic (PLEG): container finished" podID="2acce605-ed2a-4a6a-b4bd-82ddcec41594" containerID="72d3c91d46156a9e908ef780a2e4e044e7d32775b7feda69ad7969fcd6160245" exitCode=0 Oct 31 00:26:51.263046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:51.262956 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" event={"ID":"2acce605-ed2a-4a6a-b4bd-82ddcec41594","Type":"ContainerDied","Data":"72d3c91d46156a9e908ef780a2e4e044e7d32775b7feda69ad7969fcd6160245"} Oct 31 00:26:51.263046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:51.262978 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" event={"ID":"2acce605-ed2a-4a6a-b4bd-82ddcec41594","Type":"ContainerStarted","Data":"c9e7cf9b3ebf161d9d4506ea06fa4e1689c2718ad07e6bf5752a4dc164cb291b"} Oct 31 00:26:52.402020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.401998 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" Oct 31 00:26:52.730150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.730080 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2"] Oct 31 00:26:52.730460 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.730446 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2acce605-ed2a-4a6a-b4bd-82ddcec41594" containerName="curl" Oct 31 00:26:52.730507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.730462 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acce605-ed2a-4a6a-b4bd-82ddcec41594" containerName="curl" Oct 31 00:26:52.730541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.730524 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acce605-ed2a-4a6a-b4bd-82ddcec41594" containerName="curl" Oct 31 00:26:52.733552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.733536 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" Oct 31 00:26:52.741613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.741589 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2"] Oct 31 00:26:52.744934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:52.744915 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" Oct 31 00:26:53.022303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.022277 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2"] Oct 31 00:26:53.024731 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:53.024700 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a4c636a_33e4_4965_a1d9_e8641be48516.slice/crio-bd3325ff49587b79bda37a3552e8d53cc1a733127a5c475e574da7fc5da11723 WatchSource:0}: Error finding container bd3325ff49587b79bda37a3552e8d53cc1a733127a5c475e574da7fc5da11723: Status 404 returned error can't find the container with id bd3325ff49587b79bda37a3552e8d53cc1a733127a5c475e574da7fc5da11723 Oct 31 00:26:53.274850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.274760 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" Oct 31 00:26:53.274850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.274810 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683" event={"ID":"2acce605-ed2a-4a6a-b4bd-82ddcec41594","Type":"ContainerDied","Data":"c9e7cf9b3ebf161d9d4506ea06fa4e1689c2718ad07e6bf5752a4dc164cb291b"} Oct 31 00:26:53.274850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.274850 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e7cf9b3ebf161d9d4506ea06fa4e1689c2718ad07e6bf5752a4dc164cb291b" Oct 31 00:26:53.276453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.276433 5073 generic.go:356] "Generic (PLEG): container finished" podID="5a4c636a-33e4-4965-a1d9-e8641be48516" containerID="b45f5fd897bf052622bba2c2d91deab81d36987154fe910616dfa18db6b890c5" exitCode=0 Oct 31 00:26:53.276578 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.276510 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" event={"ID":"5a4c636a-33e4-4965-a1d9-e8641be48516","Type":"ContainerDied","Data":"b45f5fd897bf052622bba2c2d91deab81d36987154fe910616dfa18db6b890c5"} Oct 31 00:26:53.276578 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:53.276549 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" event={"ID":"5a4c636a-33e4-4965-a1d9-e8641be48516","Type":"ContainerStarted","Data":"bd3325ff49587b79bda37a3552e8d53cc1a733127a5c475e574da7fc5da11723"} Oct 31 00:26:54.411065 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.411042 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" Oct 31 00:26:54.849148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.849118 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d"] Oct 31 00:26:54.849486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.849472 5073 cpu_manager.go:414] "containerMap: removing container" podUID="5a4c636a-33e4-4965-a1d9-e8641be48516" containerName="curl" Oct 31 00:26:54.849537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.849487 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4c636a-33e4-4965-a1d9-e8641be48516" containerName="curl" Oct 31 00:26:54.849578 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.849568 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4c636a-33e4-4965-a1d9-e8641be48516" containerName="curl" Oct 31 00:26:54.852373 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.852358 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" Oct 31 00:26:54.860623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.860594 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d"] Oct 31 00:26:54.864028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:54.864008 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" Oct 31 00:26:55.142389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.142348 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d"] Oct 31 00:26:55.144871 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:55.144845 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod679d95f3_e935_4f31_98ec_20166a02e699.slice/crio-98b1997eca2e60580ad07539f7571bfa15500762ab02587894f750665114aa75 WatchSource:0}: Error finding container 98b1997eca2e60580ad07539f7571bfa15500762ab02587894f750665114aa75: Status 404 returned error can't find the container with id 98b1997eca2e60580ad07539f7571bfa15500762ab02587894f750665114aa75 Oct 31 00:26:55.287545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.287523 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" Oct 31 00:26:55.287673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.287561 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2" event={"ID":"5a4c636a-33e4-4965-a1d9-e8641be48516","Type":"ContainerDied","Data":"bd3325ff49587b79bda37a3552e8d53cc1a733127a5c475e574da7fc5da11723"} Oct 31 00:26:55.287673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.287596 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd3325ff49587b79bda37a3552e8d53cc1a733127a5c475e574da7fc5da11723" Oct 31 00:26:55.289257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.289236 5073 generic.go:356] "Generic (PLEG): container finished" podID="679d95f3-e935-4f31-98ec-20166a02e699" containerID="7b75fccca295683c9010015457e26146b1a8a34d61547889930d0a5d9ea1510c" exitCode=0 Oct 31 00:26:55.289378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.289272 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" event={"ID":"679d95f3-e935-4f31-98ec-20166a02e699","Type":"ContainerDied","Data":"7b75fccca295683c9010015457e26146b1a8a34d61547889930d0a5d9ea1510c"} Oct 31 00:26:55.289378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:55.289311 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" event={"ID":"679d95f3-e935-4f31-98ec-20166a02e699","Type":"ContainerStarted","Data":"98b1997eca2e60580ad07539f7571bfa15500762ab02587894f750665114aa75"} Oct 31 00:26:56.438690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.438667 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" Oct 31 00:26:56.953222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.953191 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b"] Oct 31 00:26:56.953557 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.953544 5073 cpu_manager.go:414] "containerMap: removing container" podUID="679d95f3-e935-4f31-98ec-20166a02e699" containerName="curl" Oct 31 00:26:56.953619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.953559 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679d95f3-e935-4f31-98ec-20166a02e699" containerName="curl" Oct 31 00:26:56.953655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.953640 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="679d95f3-e935-4f31-98ec-20166a02e699" containerName="curl" Oct 31 00:26:56.956452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.956436 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" Oct 31 00:26:56.964405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.964384 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b"] Oct 31 00:26:56.968769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:56.968752 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" Oct 31 00:26:57.246447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.246421 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b"] Oct 31 00:26:57.249261 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:26:57.249231 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda320a3a1_a41b_4820_9ee9_ed5abaae4e34.slice/crio-6035c285351eb21b7b9ba6c14cd5ff3dfc458b182d1ddef53d3f1c19d0044042 WatchSource:0}: Error finding container 6035c285351eb21b7b9ba6c14cd5ff3dfc458b182d1ddef53d3f1c19d0044042: Status 404 returned error can't find the container with id 6035c285351eb21b7b9ba6c14cd5ff3dfc458b182d1ddef53d3f1c19d0044042 Oct 31 00:26:57.301084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.301065 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" Oct 31 00:26:57.301192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.301108 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d" event={"ID":"679d95f3-e935-4f31-98ec-20166a02e699","Type":"ContainerDied","Data":"98b1997eca2e60580ad07539f7571bfa15500762ab02587894f750665114aa75"} Oct 31 00:26:57.301192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.301142 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b1997eca2e60580ad07539f7571bfa15500762ab02587894f750665114aa75" Oct 31 00:26:57.302661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.302639 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" event={"ID":"a320a3a1-a41b-4820-9ee9-ed5abaae4e34","Type":"ContainerStarted","Data":"e24f32aeb81ad38b299dd1b7bc8ab6e5e3adc5d5facfe4a6088bf3bd37948d7b"} Oct 31 00:26:57.302756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.302673 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" event={"ID":"a320a3a1-a41b-4820-9ee9-ed5abaae4e34","Type":"ContainerStarted","Data":"6035c285351eb21b7b9ba6c14cd5ff3dfc458b182d1ddef53d3f1c19d0044042"} Oct 31 00:26:57.320288 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:57.320248 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" podStartSLOduration=1.320238601 podStartE2EDuration="1.320238601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:57.318844021 +0000 UTC m=+1701.379618717" watchObservedRunningTime="2026-10-31 00:26:57.320238601 +0000 UTC m=+1701.381013293" Oct 31 00:26:58.308563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:58.308527 5073 generic.go:356] "Generic (PLEG): container finished" podID="a320a3a1-a41b-4820-9ee9-ed5abaae4e34" containerID="e24f32aeb81ad38b299dd1b7bc8ab6e5e3adc5d5facfe4a6088bf3bd37948d7b" exitCode=0 Oct 31 00:26:58.308960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:58.308617 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" event={"ID":"a320a3a1-a41b-4820-9ee9-ed5abaae4e34","Type":"ContainerDied","Data":"e24f32aeb81ad38b299dd1b7bc8ab6e5e3adc5d5facfe4a6088bf3bd37948d7b"} Oct 31 00:26:59.446752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:26:59.446728 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" Oct 31 00:27:00.319699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:00.319614 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" event={"ID":"a320a3a1-a41b-4820-9ee9-ed5abaae4e34","Type":"ContainerDied","Data":"6035c285351eb21b7b9ba6c14cd5ff3dfc458b182d1ddef53d3f1c19d0044042"} Oct 31 00:27:00.319699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:00.319650 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6035c285351eb21b7b9ba6c14cd5ff3dfc458b182d1ddef53d3f1c19d0044042" Oct 31 00:27:00.319699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:00.319628 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b" Oct 31 00:27:01.092405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.092374 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070"] Oct 31 00:27:01.092874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.092716 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a320a3a1-a41b-4820-9ee9-ed5abaae4e34" containerName="curl" Oct 31 00:27:01.092874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.092727 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a320a3a1-a41b-4820-9ee9-ed5abaae4e34" containerName="curl" Oct 31 00:27:01.092874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.092834 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a320a3a1-a41b-4820-9ee9-ed5abaae4e34" containerName="curl" Oct 31 00:27:01.095866 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.095848 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" Oct 31 00:27:01.104260 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.104234 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:27:01.107292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.107267 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070"] Oct 31 00:27:01.111229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.111201 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" Oct 31 00:27:01.389598 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:01.389533 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070"] Oct 31 00:27:01.393141 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:01.393114 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9124a959_d18b_4ef3_aeac_47baab79359a.slice/crio-1a4579c796b54187a6abee6961a65bc7d9f7711f032bbb102678bd3f4e73b073 WatchSource:0}: Error finding container 1a4579c796b54187a6abee6961a65bc7d9f7711f032bbb102678bd3f4e73b073: Status 404 returned error can't find the container with id 1a4579c796b54187a6abee6961a65bc7d9f7711f032bbb102678bd3f4e73b073 Oct 31 00:27:02.330762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:02.330731 5073 generic.go:356] "Generic (PLEG): container finished" podID="9124a959-d18b-4ef3-aeac-47baab79359a" containerID="22d871307738030f4913849499f0496b4b7cd5801e95f2d9c3ea3b046efe9a94" exitCode=0 Oct 31 00:27:02.331164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:02.330826 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" event={"ID":"9124a959-d18b-4ef3-aeac-47baab79359a","Type":"ContainerDied","Data":"22d871307738030f4913849499f0496b4b7cd5801e95f2d9c3ea3b046efe9a94"} Oct 31 00:27:02.331164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:02.330864 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" event={"ID":"9124a959-d18b-4ef3-aeac-47baab79359a","Type":"ContainerStarted","Data":"1a4579c796b54187a6abee6961a65bc7d9f7711f032bbb102678bd3f4e73b073"} Oct 31 00:27:03.470955 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:03.470934 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" Oct 31 00:27:04.341807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:04.341709 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" Oct 31 00:27:04.341807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:04.341752 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070" event={"ID":"9124a959-d18b-4ef3-aeac-47baab79359a","Type":"ContainerDied","Data":"1a4579c796b54187a6abee6961a65bc7d9f7711f032bbb102678bd3f4e73b073"} Oct 31 00:27:04.341982 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:04.341815 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4579c796b54187a6abee6961a65bc7d9f7711f032bbb102678bd3f4e73b073" Oct 31 00:27:05.213714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.213682 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3"] Oct 31 00:27:05.214106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.214091 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9124a959-d18b-4ef3-aeac-47baab79359a" containerName="curl" Oct 31 00:27:05.214166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.214108 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9124a959-d18b-4ef3-aeac-47baab79359a" containerName="curl" Oct 31 00:27:05.214201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.214172 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9124a959-d18b-4ef3-aeac-47baab79359a" containerName="curl" Oct 31 00:27:05.217132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.217117 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" Oct 31 00:27:05.223773 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.223755 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:27:05.225069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.225044 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3"] Oct 31 00:27:05.229730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.229714 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" Oct 31 00:27:05.507363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:05.507339 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3"] Oct 31 00:27:05.509867 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:05.509838 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84c58d8b_c0f3_44fc_bfaa_b83acb74e599.slice/crio-2adc6608b83532e8694e1625cfd8194224271e8cd27b615658b4d6a263eb9292 WatchSource:0}: Error finding container 2adc6608b83532e8694e1625cfd8194224271e8cd27b615658b4d6a263eb9292: Status 404 returned error can't find the container with id 2adc6608b83532e8694e1625cfd8194224271e8cd27b615658b4d6a263eb9292 Oct 31 00:27:06.359406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:06.359359 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" event={"ID":"84c58d8b-c0f3-44fc-bfaa-b83acb74e599","Type":"ContainerStarted","Data":"b271d23a4996a1fa09cd0a4ab710971d91667e835d9507a8d8587cf5c525d095"} Oct 31 00:27:06.359838 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:06.359415 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" event={"ID":"84c58d8b-c0f3-44fc-bfaa-b83acb74e599","Type":"ContainerStarted","Data":"2adc6608b83532e8694e1625cfd8194224271e8cd27b615658b4d6a263eb9292"} Oct 31 00:27:06.377128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:06.377077 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" podStartSLOduration=1.377061602 podStartE2EDuration="1.377061602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:06.376672768 +0000 UTC m=+1710.437447463" watchObservedRunningTime="2026-10-31 00:27:06.377061602 +0000 UTC m=+1710.437836298" Oct 31 00:27:13.085226 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:27:13.085193 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:27:13.395744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:13.395671 5073 generic.go:356] "Generic (PLEG): container finished" podID="84c58d8b-c0f3-44fc-bfaa-b83acb74e599" containerID="b271d23a4996a1fa09cd0a4ab710971d91667e835d9507a8d8587cf5c525d095" exitCode=0 Oct 31 00:27:13.395897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:13.395745 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" event={"ID":"84c58d8b-c0f3-44fc-bfaa-b83acb74e599","Type":"ContainerDied","Data":"b271d23a4996a1fa09cd0a4ab710971d91667e835d9507a8d8587cf5c525d095"} Oct 31 00:27:14.529870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:14.529846 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" Oct 31 00:27:15.407039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.407004 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" event={"ID":"84c58d8b-c0f3-44fc-bfaa-b83acb74e599","Type":"ContainerDied","Data":"2adc6608b83532e8694e1625cfd8194224271e8cd27b615658b4d6a263eb9292"} Oct 31 00:27:15.407039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.407034 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3" Oct 31 00:27:15.407272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.407047 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2adc6608b83532e8694e1625cfd8194224271e8cd27b615658b4d6a263eb9292" Oct 31 00:27:15.411146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.411120 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a"] Oct 31 00:27:15.411473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.411461 5073 cpu_manager.go:414] "containerMap: removing container" podUID="84c58d8b-c0f3-44fc-bfaa-b83acb74e599" containerName="curl" Oct 31 00:27:15.411515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.411475 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c58d8b-c0f3-44fc-bfaa-b83acb74e599" containerName="curl" Oct 31 00:27:15.411554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.411545 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c58d8b-c0f3-44fc-bfaa-b83acb74e599" containerName="curl" Oct 31 00:27:15.414562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.414539 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" Oct 31 00:27:15.419888 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.419864 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:27:15.425220 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.425200 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" Oct 31 00:27:15.439420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.439401 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a"] Oct 31 00:27:15.701715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:15.701689 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a"] Oct 31 00:27:15.704088 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:15.704061 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45affcf8_384a_425b_9210_b14c3ce700ee.slice/crio-b543c18c6118867210099d66881987097a53e5a0ce5d4a7cd7c85adeab45d59e WatchSource:0}: Error finding container b543c18c6118867210099d66881987097a53e5a0ce5d4a7cd7c85adeab45d59e: Status 404 returned error can't find the container with id b543c18c6118867210099d66881987097a53e5a0ce5d4a7cd7c85adeab45d59e Oct 31 00:27:16.413702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:16.413667 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" event={"ID":"45affcf8-384a-425b-9210-b14c3ce700ee","Type":"ContainerStarted","Data":"f5610b97b2dd508b6039e01d6ccec08308f968e9b90aa5bec1e29e176fd2d095"} Oct 31 00:27:16.413900 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:16.413714 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" event={"ID":"45affcf8-384a-425b-9210-b14c3ce700ee","Type":"ContainerStarted","Data":"b543c18c6118867210099d66881987097a53e5a0ce5d4a7cd7c85adeab45d59e"} Oct 31 00:27:16.431149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:16.431102 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" podStartSLOduration=1.43108968 podStartE2EDuration="1.43108968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:16.429926627 +0000 UTC m=+1720.490701323" watchObservedRunningTime="2026-10-31 00:27:16.43108968 +0000 UTC m=+1720.491864375" Oct 31 00:27:23.452457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:23.452427 5073 generic.go:356] "Generic (PLEG): container finished" podID="45affcf8-384a-425b-9210-b14c3ce700ee" containerID="f5610b97b2dd508b6039e01d6ccec08308f968e9b90aa5bec1e29e176fd2d095" exitCode=0 Oct 31 00:27:23.452935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:23.452506 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" event={"ID":"45affcf8-384a-425b-9210-b14c3ce700ee","Type":"ContainerDied","Data":"f5610b97b2dd508b6039e01d6ccec08308f968e9b90aa5bec1e29e176fd2d095"} Oct 31 00:27:24.584671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:24.584643 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" Oct 31 00:27:25.464599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.464568 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" Oct 31 00:27:25.464599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.464580 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a" event={"ID":"45affcf8-384a-425b-9210-b14c3ce700ee","Type":"ContainerDied","Data":"b543c18c6118867210099d66881987097a53e5a0ce5d4a7cd7c85adeab45d59e"} Oct 31 00:27:25.464843 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.464612 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b543c18c6118867210099d66881987097a53e5a0ce5d4a7cd7c85adeab45d59e" Oct 31 00:27:25.590251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.590213 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b"] Oct 31 00:27:25.590584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.590576 5073 cpu_manager.go:414] "containerMap: removing container" podUID="45affcf8-384a-425b-9210-b14c3ce700ee" containerName="curl" Oct 31 00:27:25.590624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.590587 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45affcf8-384a-425b-9210-b14c3ce700ee" containerName="curl" Oct 31 00:27:25.590658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.590649 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="45affcf8-384a-425b-9210-b14c3ce700ee" containerName="curl" Oct 31 00:27:25.593528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.593512 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" Oct 31 00:27:25.599302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.599256 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:27:25.602048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.602026 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b"] Oct 31 00:27:25.604754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.604733 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" Oct 31 00:27:25.886243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:25.886028 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b"] Oct 31 00:27:25.888488 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:25.888461 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda127a058_c4ac_4c64_a0a4_6b5fc0f64b89.slice/crio-7b93458774c7cb63f05523f75ddd8c7f8f724f276b7225e283db3616a327a4c9 WatchSource:0}: Error finding container 7b93458774c7cb63f05523f75ddd8c7f8f724f276b7225e283db3616a327a4c9: Status 404 returned error can't find the container with id 7b93458774c7cb63f05523f75ddd8c7f8f724f276b7225e283db3616a327a4c9 Oct 31 00:27:26.471432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:26.471398 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" event={"ID":"a127a058-c4ac-4c64-a0a4-6b5fc0f64b89","Type":"ContainerStarted","Data":"1e279ff2317677df1cc816e1d52656e40c376d316d3997411098c4f05b141d1f"} Oct 31 00:27:26.471631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:26.471449 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" event={"ID":"a127a058-c4ac-4c64-a0a4-6b5fc0f64b89","Type":"ContainerStarted","Data":"7b93458774c7cb63f05523f75ddd8c7f8f724f276b7225e283db3616a327a4c9"} Oct 31 00:27:26.491088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:26.491040 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" podStartSLOduration=1.491026407 podStartE2EDuration="1.491026407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:26.48924099 +0000 UTC m=+1730.550015687" watchObservedRunningTime="2026-10-31 00:27:26.491026407 +0000 UTC m=+1730.551801099" Oct 31 00:27:33.508050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:33.508023 5073 generic.go:356] "Generic (PLEG): container finished" podID="a127a058-c4ac-4c64-a0a4-6b5fc0f64b89" containerID="1e279ff2317677df1cc816e1d52656e40c376d316d3997411098c4f05b141d1f" exitCode=0 Oct 31 00:27:33.508619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:33.508100 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" event={"ID":"a127a058-c4ac-4c64-a0a4-6b5fc0f64b89","Type":"ContainerDied","Data":"1e279ff2317677df1cc816e1d52656e40c376d316d3997411098c4f05b141d1f"} Oct 31 00:27:34.652081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:34.652058 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" Oct 31 00:27:35.519456 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.519429 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" Oct 31 00:27:35.519456 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.519439 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b" event={"ID":"a127a058-c4ac-4c64-a0a4-6b5fc0f64b89","Type":"ContainerDied","Data":"7b93458774c7cb63f05523f75ddd8c7f8f724f276b7225e283db3616a327a4c9"} Oct 31 00:27:35.519663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.519472 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b93458774c7cb63f05523f75ddd8c7f8f724f276b7225e283db3616a327a4c9" Oct 31 00:27:35.784603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.784532 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32"] Oct 31 00:27:35.784979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.784908 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a127a058-c4ac-4c64-a0a4-6b5fc0f64b89" containerName="curl" Oct 31 00:27:35.784979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.784919 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a127a058-c4ac-4c64-a0a4-6b5fc0f64b89" containerName="curl" Oct 31 00:27:35.785050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.784998 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a127a058-c4ac-4c64-a0a4-6b5fc0f64b89" containerName="curl" Oct 31 00:27:35.787938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.787919 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" Oct 31 00:27:35.794731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.794709 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-network-policy-18e37582719b770d\"/\"default-dockercfg-8vxp9\"" Oct 31 00:27:35.795695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.795673 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32"] Oct 31 00:27:35.799556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:35.799535 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" Oct 31 00:27:36.075565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:36.075541 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32"] Oct 31 00:27:36.077890 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:36.077856 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40843a89_e486_42ab_bf20_b3d3d3aac3cb.slice/crio-00afd292af7474d0d745de8fe2fe0309cc1d99977bd849648563b7a683ba92c9 WatchSource:0}: Error finding container 00afd292af7474d0d745de8fe2fe0309cc1d99977bd849648563b7a683ba92c9: Status 404 returned error can't find the container with id 00afd292af7474d0d745de8fe2fe0309cc1d99977bd849648563b7a683ba92c9 Oct 31 00:27:36.525876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:36.525829 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" event={"ID":"40843a89-e486-42ab-bf20-b3d3d3aac3cb","Type":"ContainerStarted","Data":"781b7418f3f0ef2390828544e4a637e1bf8f3a6d4802bd27b6638f5b0aef2bb9"} Oct 31 00:27:36.526051 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:36.525889 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" event={"ID":"40843a89-e486-42ab-bf20-b3d3d3aac3cb","Type":"ContainerStarted","Data":"00afd292af7474d0d745de8fe2fe0309cc1d99977bd849648563b7a683ba92c9"} Oct 31 00:27:36.546618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:36.546573 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" podStartSLOduration=1.5465597070000001 podStartE2EDuration="1.546559707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:36.545946655 +0000 UTC m=+1740.606721351" watchObservedRunningTime="2026-10-31 00:27:36.546559707 +0000 UTC m=+1740.607334453" Oct 31 00:27:43.085145 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:27:43.085108 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:27:44.568504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:44.568471 5073 generic.go:356] "Generic (PLEG): container finished" podID="40843a89-e486-42ab-bf20-b3d3d3aac3cb" containerID="781b7418f3f0ef2390828544e4a637e1bf8f3a6d4802bd27b6638f5b0aef2bb9" exitCode=0 Oct 31 00:27:44.568926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:44.568542 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" event={"ID":"40843a89-e486-42ab-bf20-b3d3d3aac3cb","Type":"ContainerDied","Data":"781b7418f3f0ef2390828544e4a637e1bf8f3a6d4802bd27b6638f5b0aef2bb9"} Oct 31 00:27:45.702237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.702205 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" Oct 31 00:27:45.984041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.984012 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75"] Oct 31 00:27:45.984418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.984403 5073 cpu_manager.go:414] "containerMap: removing container" podUID="40843a89-e486-42ab-bf20-b3d3d3aac3cb" containerName="curl" Oct 31 00:27:45.984466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.984420 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40843a89-e486-42ab-bf20-b3d3d3aac3cb" containerName="curl" Oct 31 00:27:45.984516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.984507 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="40843a89-e486-42ab-bf20-b3d3d3aac3cb" containerName="curl" Oct 31 00:27:45.987424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.987406 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" Oct 31 00:27:45.996480 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.996455 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75"] Oct 31 00:27:45.999473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:45.999453 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" Oct 31 00:27:46.276820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.276773 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75"] Oct 31 00:27:46.279156 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:46.279129 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf415833c_68c3_42fc_8db7_69c63f713307.slice/crio-8d01b599dacf6ba923de692d23abab1f11d4132a17748221ccadec73b33d75be WatchSource:0}: Error finding container 8d01b599dacf6ba923de692d23abab1f11d4132a17748221ccadec73b33d75be: Status 404 returned error can't find the container with id 8d01b599dacf6ba923de692d23abab1f11d4132a17748221ccadec73b33d75be Oct 31 00:27:46.581175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.581106 5073 generic.go:356] "Generic (PLEG): container finished" podID="f415833c-68c3-42fc-8db7-69c63f713307" containerID="80912767e5ddfa6376f475714eee5e255875147ac4f18d56b0b6570157567991" exitCode=0 Oct 31 00:27:46.581175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.581156 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" event={"ID":"f415833c-68c3-42fc-8db7-69c63f713307","Type":"ContainerDied","Data":"80912767e5ddfa6376f475714eee5e255875147ac4f18d56b0b6570157567991"} Oct 31 00:27:46.581358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.581206 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" event={"ID":"f415833c-68c3-42fc-8db7-69c63f713307","Type":"ContainerStarted","Data":"8d01b599dacf6ba923de692d23abab1f11d4132a17748221ccadec73b33d75be"} Oct 31 00:27:46.582675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.582646 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" event={"ID":"40843a89-e486-42ab-bf20-b3d3d3aac3cb","Type":"ContainerDied","Data":"00afd292af7474d0d745de8fe2fe0309cc1d99977bd849648563b7a683ba92c9"} Oct 31 00:27:46.582675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.582679 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00afd292af7474d0d745de8fe2fe0309cc1d99977bd849648563b7a683ba92c9" Oct 31 00:27:46.582848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:46.582677 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32" Oct 31 00:27:47.715436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:47.715407 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" Oct 31 00:27:48.103397 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.103364 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4"] Oct 31 00:27:48.103707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.103694 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f415833c-68c3-42fc-8db7-69c63f713307" containerName="curl" Oct 31 00:27:48.103758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.103708 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f415833c-68c3-42fc-8db7-69c63f713307" containerName="curl" Oct 31 00:27:48.103821 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.103779 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f415833c-68c3-42fc-8db7-69c63f713307" containerName="curl" Oct 31 00:27:48.106623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.106606 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" Oct 31 00:27:48.114122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.114099 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4"] Oct 31 00:27:48.118709 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.118688 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" Oct 31 00:27:48.397864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.397839 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4"] Oct 31 00:27:48.400313 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:48.400286 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba2beafd_96cf_4ed6_9fc8_a5f1564d039f.slice/crio-bf1fbad6b5143b96494835c945723c76d673595e644f6fdf3451bbe898d055e2 WatchSource:0}: Error finding container bf1fbad6b5143b96494835c945723c76d673595e644f6fdf3451bbe898d055e2: Status 404 returned error can't find the container with id bf1fbad6b5143b96494835c945723c76d673595e644f6fdf3451bbe898d055e2 Oct 31 00:27:48.594515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.594487 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" Oct 31 00:27:48.594654 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.594529 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75" event={"ID":"f415833c-68c3-42fc-8db7-69c63f713307","Type":"ContainerDied","Data":"8d01b599dacf6ba923de692d23abab1f11d4132a17748221ccadec73b33d75be"} Oct 31 00:27:48.594654 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.594568 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d01b599dacf6ba923de692d23abab1f11d4132a17748221ccadec73b33d75be" Oct 31 00:27:48.596221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.596197 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" event={"ID":"ba2beafd-96cf-4ed6-9fc8-a5f1564d039f","Type":"ContainerStarted","Data":"88c4da1c36677edf90b6d18310851dda19960ec7c433ce170efc8510bd087831"} Oct 31 00:27:48.596362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.596233 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" event={"ID":"ba2beafd-96cf-4ed6-9fc8-a5f1564d039f","Type":"ContainerStarted","Data":"bf1fbad6b5143b96494835c945723c76d673595e644f6fdf3451bbe898d055e2"} Oct 31 00:27:48.614310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:48.614272 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" podStartSLOduration=0.614261021 podStartE2EDuration="614.261021ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:48.612543324 +0000 UTC m=+1752.673318022" watchObservedRunningTime="2026-10-31 00:27:48.614261021 +0000 UTC m=+1752.675035713" Oct 31 00:27:53.624353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:53.624317 5073 generic.go:356] "Generic (PLEG): container finished" podID="ba2beafd-96cf-4ed6-9fc8-a5f1564d039f" containerID="88c4da1c36677edf90b6d18310851dda19960ec7c433ce170efc8510bd087831" exitCode=0 Oct 31 00:27:53.624710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:53.624393 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" event={"ID":"ba2beafd-96cf-4ed6-9fc8-a5f1564d039f","Type":"ContainerDied","Data":"88c4da1c36677edf90b6d18310851dda19960ec7c433ce170efc8510bd087831"} Oct 31 00:27:54.773911 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:54.773890 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" Oct 31 00:27:55.635898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:55.635862 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" event={"ID":"ba2beafd-96cf-4ed6-9fc8-a5f1564d039f","Type":"ContainerDied","Data":"bf1fbad6b5143b96494835c945723c76d673595e644f6fdf3451bbe898d055e2"} Oct 31 00:27:55.635898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:55.635900 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1fbad6b5143b96494835c945723c76d673595e644f6fdf3451bbe898d055e2" Oct 31 00:27:55.635898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:55.635877 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4" Oct 31 00:27:56.286876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.286841 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318"] Oct 31 00:27:56.287387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.287207 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ba2beafd-96cf-4ed6-9fc8-a5f1564d039f" containerName="curl" Oct 31 00:27:56.287387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.287218 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2beafd-96cf-4ed6-9fc8-a5f1564d039f" containerName="curl" Oct 31 00:27:56.287387 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.287275 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2beafd-96cf-4ed6-9fc8-a5f1564d039f" containerName="curl" Oct 31 00:27:56.290397 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.290377 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:56.297687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.297654 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:27:56.299979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.299956 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318"] Oct 31 00:27:56.408139 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.408102 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate\") pod \"targetallocator-mtls-allowed-18e375b03098d318\" (UID: \"d3b75e5a-c794-4d68-9f13-d546e9fa1d85\") " pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:56.509018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.508993 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate\") pod \"targetallocator-mtls-allowed-18e375b03098d318\" (UID: \"d3b75e5a-c794-4d68-9f13-d546e9fa1d85\") " pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:56.511674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.511654 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate\") pod \"targetallocator-mtls-allowed-18e375b03098d318\" (UID: \"d3b75e5a-c794-4d68-9f13-d546e9fa1d85\") " pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:56.602721 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.602656 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:56.879948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:56.879880 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318"] Oct 31 00:27:56.883380 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:27:56.883348 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3b75e5a_c794_4d68_9f13_d546e9fa1d85.slice/crio-81c1260ff07b379d126c4798022ab6fdf9a4401fbf60f2251bf0faefa2bc9d69 WatchSource:0}: Error finding container 81c1260ff07b379d126c4798022ab6fdf9a4401fbf60f2251bf0faefa2bc9d69: Status 404 returned error can't find the container with id 81c1260ff07b379d126c4798022ab6fdf9a4401fbf60f2251bf0faefa2bc9d69 Oct 31 00:27:57.647221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:57.647190 5073 generic.go:356] "Generic (PLEG): container finished" podID="d3b75e5a-c794-4d68-9f13-d546e9fa1d85" containerID="38c31d54de1ff7d935a6d29eb1cc93588f061054f44288b742cb5964b4c52bca" exitCode=0 Oct 31 00:27:57.647618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:57.647280 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" event={"ID":"d3b75e5a-c794-4d68-9f13-d546e9fa1d85","Type":"ContainerDied","Data":"38c31d54de1ff7d935a6d29eb1cc93588f061054f44288b742cb5964b4c52bca"} Oct 31 00:27:57.647618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:57.647328 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" event={"ID":"d3b75e5a-c794-4d68-9f13-d546e9fa1d85","Type":"ContainerStarted","Data":"81c1260ff07b379d126c4798022ab6fdf9a4401fbf60f2251bf0faefa2bc9d69"} Oct 31 00:27:58.786927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:58.786899 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:58.823778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:58.823752 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate\" (UniqueName: \"kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate\") pod \"d3b75e5a-c794-4d68-9f13-d546e9fa1d85\" (UID: \"d3b75e5a-c794-4d68-9f13-d546e9fa1d85\") " Oct 31 00:27:58.825834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:58.825812 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate" pod "d3b75e5a-c794-4d68-9f13-d546e9fa1d85" (UID: "d3b75e5a-c794-4d68-9f13-d546e9fa1d85"). InnerVolumeSpecName "client-certificate". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:58.924375 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:58.924322 5073 reconciler_common.go:299] "Volume detached for volume \"client-certificate\" (UniqueName: \"kubernetes.io/secret/d3b75e5a-c794-4d68-9f13-d546e9fa1d85-client-certificate\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:59.659907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:59.659875 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" Oct 31 00:27:59.660080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:59.659907 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318" event={"ID":"d3b75e5a-c794-4d68-9f13-d546e9fa1d85","Type":"ContainerDied","Data":"81c1260ff07b379d126c4798022ab6fdf9a4401fbf60f2251bf0faefa2bc9d69"} Oct 31 00:27:59.660080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:27:59.659951 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c1260ff07b379d126c4798022ab6fdf9a4401fbf60f2251bf0faefa2bc9d69" Oct 31 00:28:00.415631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.415591 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751"] Oct 31 00:28:00.416044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.415950 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d3b75e5a-c794-4d68-9f13-d546e9fa1d85" containerName="curl" Oct 31 00:28:00.416044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.415962 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b75e5a-c794-4d68-9f13-d546e9fa1d85" containerName="curl" Oct 31 00:28:00.416044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.416041 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b75e5a-c794-4d68-9f13-d546e9fa1d85" containerName="curl" Oct 31 00:28:00.419001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.418985 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" Oct 31 00:28:00.427524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.427497 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751"] Oct 31 00:28:00.430449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.430431 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" Oct 31 00:28:00.711021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:00.710977 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751"] Oct 31 00:28:00.713236 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:00.713209 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d8502ae_6ada_43ab_982f_41d9525af090.slice/crio-94160c137bb4d0959fc112a11fdf86b5626f799a040f67758eee559d5310a42b WatchSource:0}: Error finding container 94160c137bb4d0959fc112a11fdf86b5626f799a040f67758eee559d5310a42b: Status 404 returned error can't find the container with id 94160c137bb4d0959fc112a11fdf86b5626f799a040f67758eee559d5310a42b Oct 31 00:28:01.672191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:01.672155 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" event={"ID":"7d8502ae-6ada-43ab-982f-41d9525af090","Type":"ContainerStarted","Data":"76fed8aaf65c10967069d2245b5f4e6fbd6fd4b6e732d9d9b6ab95fc08ae9017"} Oct 31 00:28:01.672584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:01.672200 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" event={"ID":"7d8502ae-6ada-43ab-982f-41d9525af090","Type":"ContainerStarted","Data":"94160c137bb4d0959fc112a11fdf86b5626f799a040f67758eee559d5310a42b"} Oct 31 00:28:01.690005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:01.689953 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" podStartSLOduration=1.689937142 podStartE2EDuration="1.689937142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:01.689688775 +0000 UTC m=+1765.750463470" watchObservedRunningTime="2026-10-31 00:28:01.689937142 +0000 UTC m=+1765.750711835" Oct 31 00:28:06.701313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:06.701275 5073 generic.go:356] "Generic (PLEG): container finished" podID="7d8502ae-6ada-43ab-982f-41d9525af090" containerID="76fed8aaf65c10967069d2245b5f4e6fbd6fd4b6e732d9d9b6ab95fc08ae9017" exitCode=0 Oct 31 00:28:06.701712 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:06.701348 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" event={"ID":"7d8502ae-6ada-43ab-982f-41d9525af090","Type":"ContainerDied","Data":"76fed8aaf65c10967069d2245b5f4e6fbd6fd4b6e732d9d9b6ab95fc08ae9017"} Oct 31 00:28:07.844062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:07.844040 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" Oct 31 00:28:08.525735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.525705 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751"] Oct 31 00:28:08.530713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.530689 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751"] Oct 31 00:28:08.552511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.552485 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318"] Oct 31 00:28:08.557494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.557474 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-mtls-allowed-18e375b03098d318"] Oct 31 00:28:08.579599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.579579 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4"] Oct 31 00:28:08.584418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.584394 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-untrusted-denied-18e375ae4812bdd4"] Oct 31 00:28:08.606973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.606942 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75"] Oct 31 00:28:08.611339 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.611318 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/targetallocator-collector-allowed-18e375adc969ee75"] Oct 31 00:28:08.633621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.633600 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32"] Oct 31 00:28:08.640547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.640530 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-wrong-role-denied-18e375ab69bb4b32"] Oct 31 00:28:08.663947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.663924 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b"] Oct 31 00:28:08.669952 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.669930 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-role-only-denied-18e375a90b147d7b"] Oct 31 00:28:08.692060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.692041 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a"] Oct 31 00:28:08.696749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.696729 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-part-only-denied-18e375a6ab424c8a"] Oct 31 00:28:08.713338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.713323 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94160c137bb4d0959fc112a11fdf86b5626f799a040f67758eee559d5310a42b" Oct 31 00:28:08.713338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.713330 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" Oct 31 00:28:08.717983 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:08.717951 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"targetallocator-mtls-untrusted-denied-18e375b126af5751\" is forbidden: User \"system:node:ip-10-0-140-246.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"redhat-ods-monitoring\": no relationship found between node 'ip-10-0-140-246.us-east-2.compute.internal' and this object" podUID="7d8502ae-6ada-43ab-982f-41d9525af090" pod="redhat-ods-monitoring/targetallocator-mtls-untrusted-denied-18e375b126af5751" Oct 31 00:28:08.719136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.719119 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3"] Oct 31 00:28:08.723977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.723959 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-unlabeled-denied-18e375a44ca7cbb3"] Oct 31 00:28:08.746134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.746108 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070"] Oct 31 00:28:08.751730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.751702 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-inferenceservice-allowed-18e375a355fa4070"] Oct 31 00:28:08.772946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.772927 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b"] Oct 31 00:28:08.777534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.777490 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-scheduler-allowed-18e375a2604cd21b"] Oct 31 00:28:08.799561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.799539 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d"] Oct 31 00:28:08.806701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.806669 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-prefill-allowed-18e375a1e1f79e0d"] Oct 31 00:28:08.829144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.829121 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2"] Oct 31 00:28:08.833868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.833849 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-prefill-allowed-18e375a1638152f2"] Oct 31 00:28:08.856182 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.856159 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683"] Oct 31 00:28:08.860965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.860939 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-leader-allowed-18e375a0e5d41683"] Oct 31 00:28:08.883334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.883315 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3"] Oct 31 00:28:08.888135 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.888116 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-worker-allowed-18e375a06787f5b3"] Oct 31 00:28:08.910139 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.910114 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa"] Oct 31 00:28:08.915083 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.915064 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-prefill-allowed-18e3759fe916e8fa"] Oct 31 00:28:08.936689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.936671 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550"] Oct 31 00:28:08.945833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.945815 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-network-policy-18e37582719b770d/collector-otlp-workload-allowed-18e3759ef30f2550"] Oct 31 00:28:08.967698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.967678 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267"] Oct 31 00:28:08.972327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.972311 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-wrong-role-denied-18e3759c925e4267"] Oct 31 00:28:08.994091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.994063 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10"] Oct 31 00:28:08.998605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:08.998586 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-role-only-denied-18e3759a32c15d10"] Oct 31 00:28:09.020628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.020605 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658"] Oct 31 00:28:09.025110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.025093 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-part-only-denied-18e37597d3206658"] Oct 31 00:28:09.048303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.048250 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98"] Oct 31 00:28:09.053291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.053271 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-unlabeled-denied-18e3759573b4fa98"] Oct 31 00:28:09.075070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.075045 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1"] Oct 31 00:28:09.080017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.079996 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-inferenceservice-allowed-18e375931405a1b1"] Oct 31 00:28:09.101840 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.101813 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6"] Oct 31 00:28:09.106538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.106520 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-scheduler-allowed-18e37590b46124f6"] Oct 31 00:28:09.128703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.128673 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64"] Oct 31 00:28:09.135382 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.135365 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-prefill-allowed-18e3758e54baec64"] Oct 31 00:28:09.157097 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.157077 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa"] Oct 31 00:28:09.162947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.162924 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-prefill-allowed-18e3758bf4e422aa"] Oct 31 00:28:09.183975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.183956 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903"] Oct 31 00:28:09.189253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.189234 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-leader-allowed-18e375899555f903"] Oct 31 00:28:09.211203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.211184 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e"] Oct 31 00:28:09.215592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.215574 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-worker-allowed-18e3758735c5395e"] Oct 31 00:28:09.237616 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.237598 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d"] Oct 31 00:28:09.242351 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.242332 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-prefill-allowed-18e37584d663e91d"] Oct 31 00:28:09.263816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.263775 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba"] Oct 31 00:28:09.266351 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.266334 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e3758273f237ba"] Oct 31 00:28:09.309071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.309013 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f"] Oct 31 00:28:09.313659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:09.313641 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e3757f2074bb8f"] Oct 31 00:28:10.058187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.058148 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d7df63-e32e-4955-964e-2ed4aa29cc2a" path="/var/lib/kubelet/pods/18d7df63-e32e-4955-964e-2ed4aa29cc2a/volumes" Oct 31 00:28:10.058684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.058486 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b8c5f4-d54c-4900-b174-ee3c499a5387" path="/var/lib/kubelet/pods/27b8c5f4-d54c-4900-b174-ee3c499a5387/volumes" Oct 31 00:28:10.058821 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.058778 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acce605-ed2a-4a6a-b4bd-82ddcec41594" path="/var/lib/kubelet/pods/2acce605-ed2a-4a6a-b4bd-82ddcec41594/volumes" Oct 31 00:28:10.059072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.059056 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40843a89-e486-42ab-bf20-b3d3d3aac3cb" path="/var/lib/kubelet/pods/40843a89-e486-42ab-bf20-b3d3d3aac3cb/volumes" Oct 31 00:28:10.059299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.059283 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45affcf8-384a-425b-9210-b14c3ce700ee" path="/var/lib/kubelet/pods/45affcf8-384a-425b-9210-b14c3ce700ee/volumes" Oct 31 00:28:10.059872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.059852 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4c636a-33e4-4965-a1d9-e8641be48516" path="/var/lib/kubelet/pods/5a4c636a-33e4-4965-a1d9-e8641be48516/volumes" Oct 31 00:28:10.060117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.060100 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e96b98-2e5e-4af4-acf9-87bfd9f221bb" path="/var/lib/kubelet/pods/65e96b98-2e5e-4af4-acf9-87bfd9f221bb/volumes" Oct 31 00:28:10.060344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.060328 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67187bc6-e774-4aba-a8d0-49dd42a91fa4" path="/var/lib/kubelet/pods/67187bc6-e774-4aba-a8d0-49dd42a91fa4/volumes" Oct 31 00:28:10.061191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.061163 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679d95f3-e935-4f31-98ec-20166a02e699" path="/var/lib/kubelet/pods/679d95f3-e935-4f31-98ec-20166a02e699/volumes" Oct 31 00:28:10.061528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.061507 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea59b23-25b1-40bb-b02b-d8993605f0c5" path="/var/lib/kubelet/pods/6ea59b23-25b1-40bb-b02b-d8993605f0c5/volumes" Oct 31 00:28:10.062344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.062320 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7248d8fd-b22a-4dbe-8544-4adeaabf640f" path="/var/lib/kubelet/pods/7248d8fd-b22a-4dbe-8544-4adeaabf640f/volumes" Oct 31 00:28:10.062680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.062659 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8502ae-6ada-43ab-982f-41d9525af090" path="/var/lib/kubelet/pods/7d8502ae-6ada-43ab-982f-41d9525af090/volumes" Oct 31 00:28:10.063026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.063005 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828687c7-778b-4e04-8bc9-f73a509344c2" path="/var/lib/kubelet/pods/828687c7-778b-4e04-8bc9-f73a509344c2/volumes" Oct 31 00:28:10.063390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.063370 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c58d8b-c0f3-44fc-bfaa-b83acb74e599" path="/var/lib/kubelet/pods/84c58d8b-c0f3-44fc-bfaa-b83acb74e599/volumes" Oct 31 00:28:10.063771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.063749 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29" path="/var/lib/kubelet/pods/8b4e7b4f-86e7-44d5-a1ca-d605bebe2d29/volumes" Oct 31 00:28:10.064550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.064523 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9124a959-d18b-4ef3-aeac-47baab79359a" path="/var/lib/kubelet/pods/9124a959-d18b-4ef3-aeac-47baab79359a/volumes" Oct 31 00:28:10.064985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.064965 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c58f06-681f-4a0a-8c92-969c42255072" path="/var/lib/kubelet/pods/97c58f06-681f-4a0a-8c92-969c42255072/volumes" Oct 31 00:28:10.065338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.065317 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a127a058-c4ac-4c64-a0a4-6b5fc0f64b89" path="/var/lib/kubelet/pods/a127a058-c4ac-4c64-a0a4-6b5fc0f64b89/volumes" Oct 31 00:28:10.065689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.065666 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a320a3a1-a41b-4820-9ee9-ed5abaae4e34" path="/var/lib/kubelet/pods/a320a3a1-a41b-4820-9ee9-ed5abaae4e34/volumes" Oct 31 00:28:10.066079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.066059 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6086f36-26b5-4cd1-8a0f-3abbeec070ef" path="/var/lib/kubelet/pods/a6086f36-26b5-4cd1-8a0f-3abbeec070ef/volumes" Oct 31 00:28:10.066871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.066851 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0469e63-7fcc-4a89-b0c9-94decba395e5" path="/var/lib/kubelet/pods/b0469e63-7fcc-4a89-b0c9-94decba395e5/volumes" Oct 31 00:28:10.067222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.067197 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2beafd-96cf-4ed6-9fc8-a5f1564d039f" path="/var/lib/kubelet/pods/ba2beafd-96cf-4ed6-9fc8-a5f1564d039f/volumes" Oct 31 00:28:10.067600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.067580 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a65b56-e108-4eeb-8b2e-19bf3a824f65" path="/var/lib/kubelet/pods/c3a65b56-e108-4eeb-8b2e-19bf3a824f65/volumes" Oct 31 00:28:10.068060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.068040 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b75e5a-c794-4d68-9f13-d546e9fa1d85" path="/var/lib/kubelet/pods/d3b75e5a-c794-4d68-9f13-d546e9fa1d85/volumes" Oct 31 00:28:10.068640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.068618 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d422f5ff-1bce-44ed-8118-0e3c007891ab" path="/var/lib/kubelet/pods/d422f5ff-1bce-44ed-8118-0e3c007891ab/volumes" Oct 31 00:28:10.068965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.068950 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7d79d2-fef6-44eb-95bc-e5066b2b8567" path="/var/lib/kubelet/pods/df7d79d2-fef6-44eb-95bc-e5066b2b8567/volumes" Oct 31 00:28:10.069158 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.069147 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e093c8aa-677e-4518-af69-39ef54e79ab1" path="/var/lib/kubelet/pods/e093c8aa-677e-4518-af69-39ef54e79ab1/volumes" Oct 31 00:28:10.069347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.069336 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21e56f1-66b2-4888-8b1f-8644287b71f7" path="/var/lib/kubelet/pods/e21e56f1-66b2-4888-8b1f-8644287b71f7/volumes" Oct 31 00:28:10.069527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.069517 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f415833c-68c3-42fc-8db7-69c63f713307" path="/var/lib/kubelet/pods/f415833c-68c3-42fc-8db7-69c63f713307/volumes" Oct 31 00:28:10.393102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.393029 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593"] Oct 31 00:28:10.393370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.393357 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7d8502ae-6ada-43ab-982f-41d9525af090" containerName="curl" Oct 31 00:28:10.393423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.393371 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8502ae-6ada-43ab-982f-41d9525af090" containerName="curl" Oct 31 00:28:10.393458 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.393441 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8502ae-6ada-43ab-982f-41d9525af090" containerName="curl" Oct 31 00:28:10.397862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.397844 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" Oct 31 00:28:10.403556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.403531 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-namespace-auth-18e375b33f6816f7\"/\"default-dockercfg-b6ks5\"" Oct 31 00:28:10.404640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.404502 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593"] Oct 31 00:28:10.409071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.409047 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" Oct 31 00:28:10.682642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.682571 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593"] Oct 31 00:28:10.685696 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:10.685670 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c63f37a_52fc_4867_90b1_88605b2fd976.slice/crio-8fcea56bc4e4944edc9ad6e6a99f5f688ec79fa4035d1f39f20fc80fcc38360d WatchSource:0}: Error finding container 8fcea56bc4e4944edc9ad6e6a99f5f688ec79fa4035d1f39f20fc80fcc38360d: Status 404 returned error can't find the container with id 8fcea56bc4e4944edc9ad6e6a99f5f688ec79fa4035d1f39f20fc80fcc38360d Oct 31 00:28:10.725431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:10.725407 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" event={"ID":"6c63f37a-52fc-4867-90b1-88605b2fd976","Type":"ContainerStarted","Data":"8fcea56bc4e4944edc9ad6e6a99f5f688ec79fa4035d1f39f20fc80fcc38360d"} Oct 31 00:28:11.731916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:11.731877 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" event={"ID":"6c63f37a-52fc-4867-90b1-88605b2fd976","Type":"ContainerStarted","Data":"cd574df4f1fb37cb366e5c818ee148523d0979b15aadcfb3e009544ebe07a9d5"} Oct 31 00:28:11.748157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:11.748110 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" podStartSLOduration=1.748093956 podStartE2EDuration="1.748093956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:11.747848547 +0000 UTC m=+1775.808623244" watchObservedRunningTime="2026-10-31 00:28:11.748093956 +0000 UTC m=+1775.808868653" Oct 31 00:28:13.084953 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:13.084917 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:28:14.711320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.711281 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp"] Oct 31 00:28:14.714742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.714723 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.724181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.724160 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp"] Oct 31 00:28:14.843925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.843892 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.844061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.844011 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8ngt\" (UniqueName: \"kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.844130 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.844063 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.945089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.945053 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ngt\" (UniqueName: \"kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.945226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.945112 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.945226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.945157 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.945735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.945716 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.947522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.947500 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:14.958866 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:14.958846 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ngt\" (UniqueName: \"kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt\") pod \"data-science-collector-targetallocator-77f8cbbcbf-kchmp\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:15.027093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.027070 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:15.303601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.303426 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp"] Oct 31 00:28:15.306308 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:15.306279 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8bd94b9_84d2_442e_9401_d45ce3d16b6b.slice/crio-d3b6ba61b4dd6dfa3c3d504079e8bc1c43d5b0d24c4b27f2b38f080279c0013c WatchSource:0}: Error finding container d3b6ba61b4dd6dfa3c3d504079e8bc1c43d5b0d24c4b27f2b38f080279c0013c: Status 404 returned error can't find the container with id d3b6ba61b4dd6dfa3c3d504079e8bc1c43d5b0d24c4b27f2b38f080279c0013c Oct 31 00:28:15.308547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.308513 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:28:15.308634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.308574 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:28:15.756842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.756807 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" event={"ID":"a8bd94b9-84d2-442e-9401-d45ce3d16b6b","Type":"ContainerStarted","Data":"727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8"} Oct 31 00:28:15.757245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.756857 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" event={"ID":"a8bd94b9-84d2-442e-9401-d45ce3d16b6b","Type":"ContainerStarted","Data":"d3b6ba61b4dd6dfa3c3d504079e8bc1c43d5b0d24c4b27f2b38f080279c0013c"} Oct 31 00:28:15.757245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.757014 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:15.759132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.759110 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:15.777612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.777482 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" podStartSLOduration=1.7774672900000001 podStartE2EDuration="1.77746729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:15.776242307 +0000 UTC m=+1779.837017003" watchObservedRunningTime="2026-10-31 00:28:15.77746729 +0000 UTC m=+1779.838241985" Oct 31 00:28:15.828558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.828527 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg"] Oct 31 00:28:15.828728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:15.828707 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" podUID="af81481e-7e3c-45f8-ad60-0129f1a6bcd6" containerName="ta-container" containerID="cri-o://75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59" gracePeriod=30 Oct 31 00:28:16.077501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.077479 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:28:16.155398 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.155368 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw\" (UniqueName: \"kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw\") pod \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " Oct 31 00:28:16.155539 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.155451 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert\") pod \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " Oct 31 00:28:16.155539 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.155484 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal\" (UniqueName: \"kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal\") pod \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\" (UID: \"af81481e-7e3c-45f8-ad60-0129f1a6bcd6\") " Oct 31 00:28:16.155945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.155921 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal" pod "af81481e-7e3c-45f8-ad60-0129f1a6bcd6" (UID: "af81481e-7e3c-45f8-ad60-0129f1a6bcd6"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:16.157688 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.157659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw" pod "af81481e-7e3c-45f8-ad60-0129f1a6bcd6" (UID: "af81481e-7e3c-45f8-ad60-0129f1a6bcd6"). InnerVolumeSpecName "kube-api-access-zlckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:16.157807 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.157690 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert" pod "af81481e-7e3c-45f8-ad60-0129f1a6bcd6" (UID: "af81481e-7e3c-45f8-ad60-0129f1a6bcd6"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:16.256309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.256282 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:16.256309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.256309 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:16.256451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.256319 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlckw\" (UniqueName: \"kubernetes.io/projected/af81481e-7e3c-45f8-ad60-0129f1a6bcd6-kube-api-access-zlckw\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:16.763545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.763510 5073 generic.go:356] "Generic (PLEG): container finished" podID="af81481e-7e3c-45f8-ad60-0129f1a6bcd6" containerID="75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59" exitCode=0 Oct 31 00:28:16.764000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.763555 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" Oct 31 00:28:16.764000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.763578 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" event={"ID":"af81481e-7e3c-45f8-ad60-0129f1a6bcd6","Type":"ContainerDied","Data":"75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59"} Oct 31 00:28:16.764000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.763623 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg" event={"ID":"af81481e-7e3c-45f8-ad60-0129f1a6bcd6","Type":"ContainerDied","Data":"28d0155ef0768fe1700b9b90d3474abc72177eb5d32f90a436c142b3d84c97cf"} Oct 31 00:28:16.764000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.763643 5073 scope.go:122] "RemoveContainer" containerID="75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59" Oct 31 00:28:16.773872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.773857 5073 scope.go:122] "RemoveContainer" containerID="75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59" Oct 31 00:28:16.774150 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:16.774127 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59\": container with ID starting with 75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59 not found: ID does not exist" containerID="75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59" Oct 31 00:28:16.774203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.774158 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59"} err="failed to get container status \"75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59\": rpc error: code = NotFound desc = could not find container \"75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59\": container with ID starting with 75aa8e16d7fd5cebb3657bd607037f7974ffa93d59eb6290d42ba2cb4a0bee59 not found: ID does not exist" Oct 31 00:28:16.785850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.785829 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg"] Oct 31 00:28:16.791967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:16.791946 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-55b78f4f4d-qlzsg"] Oct 31 00:28:18.058317 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:18.058283 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af81481e-7e3c-45f8-ad60-0129f1a6bcd6" path="/var/lib/kubelet/pods/af81481e-7e3c-45f8-ad60-0129f1a6bcd6/volumes" Oct 31 00:28:18.776545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:18.776518 5073 generic.go:356] "Generic (PLEG): container finished" podID="6c63f37a-52fc-4867-90b1-88605b2fd976" containerID="cd574df4f1fb37cb366e5c818ee148523d0979b15aadcfb3e009544ebe07a9d5" exitCode=0 Oct 31 00:28:18.776678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:18.776575 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" event={"ID":"6c63f37a-52fc-4867-90b1-88605b2fd976","Type":"ContainerDied","Data":"cd574df4f1fb37cb366e5c818ee148523d0979b15aadcfb3e009544ebe07a9d5"} Oct 31 00:28:19.908902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:19.908879 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" Oct 31 00:28:20.778856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.778825 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e"] Oct 31 00:28:20.779237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.779218 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6c63f37a-52fc-4867-90b1-88605b2fd976" containerName="curl" Oct 31 00:28:20.779326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.779239 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c63f37a-52fc-4867-90b1-88605b2fd976" containerName="curl" Oct 31 00:28:20.779326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.779260 5073 cpu_manager.go:414] "containerMap: removing container" podUID="af81481e-7e3c-45f8-ad60-0129f1a6bcd6" containerName="ta-container" Oct 31 00:28:20.779326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.779268 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af81481e-7e3c-45f8-ad60-0129f1a6bcd6" containerName="ta-container" Oct 31 00:28:20.779481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.779383 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c63f37a-52fc-4867-90b1-88605b2fd976" containerName="curl" Oct 31 00:28:20.779481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.779396 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="af81481e-7e3c-45f8-ad60-0129f1a6bcd6" containerName="ta-container" Oct 31 00:28:20.782324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.782300 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" Oct 31 00:28:20.787749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.787722 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" event={"ID":"6c63f37a-52fc-4867-90b1-88605b2fd976","Type":"ContainerDied","Data":"8fcea56bc4e4944edc9ad6e6a99f5f688ec79fa4035d1f39f20fc80fcc38360d"} Oct 31 00:28:20.787857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.787762 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fcea56bc4e4944edc9ad6e6a99f5f688ec79fa4035d1f39f20fc80fcc38360d" Oct 31 00:28:20.787857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.787766 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593" Oct 31 00:28:20.790495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.790472 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e"] Oct 31 00:28:20.795747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:20.795728 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" Oct 31 00:28:21.071610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:21.071546 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e"] Oct 31 00:28:21.074669 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:21.074640 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cdf683a_1700_4276_b48d_e229a1ee79bb.slice/crio-a40bddb2de1ab4bb01ed89d6269d95c79eb761b1c862f6879abd6026eaca9dc9 WatchSource:0}: Error finding container a40bddb2de1ab4bb01ed89d6269d95c79eb761b1c862f6879abd6026eaca9dc9: Status 404 returned error can't find the container with id a40bddb2de1ab4bb01ed89d6269d95c79eb761b1c862f6879abd6026eaca9dc9 Oct 31 00:28:21.795475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:21.795434 5073 generic.go:356] "Generic (PLEG): container finished" podID="7cdf683a-1700-4276-b48d-e229a1ee79bb" containerID="5c9833fa2742ce62723a18d3b477f8a5ba3647cc9c7177746ab67b5194fedc3d" exitCode=0 Oct 31 00:28:21.795667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:21.795508 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" event={"ID":"7cdf683a-1700-4276-b48d-e229a1ee79bb","Type":"ContainerDied","Data":"5c9833fa2742ce62723a18d3b477f8a5ba3647cc9c7177746ab67b5194fedc3d"} Oct 31 00:28:21.795667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:21.795568 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" event={"ID":"7cdf683a-1700-4276-b48d-e229a1ee79bb","Type":"ContainerStarted","Data":"a40bddb2de1ab4bb01ed89d6269d95c79eb761b1c862f6879abd6026eaca9dc9"} Oct 31 00:28:22.925882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:22.925856 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" Oct 31 00:28:23.807939 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:23.807902 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" event={"ID":"7cdf683a-1700-4276-b48d-e229a1ee79bb","Type":"ContainerDied","Data":"a40bddb2de1ab4bb01ed89d6269d95c79eb761b1c862f6879abd6026eaca9dc9"} Oct 31 00:28:23.808119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:23.807951 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40bddb2de1ab4bb01ed89d6269d95c79eb761b1c862f6879abd6026eaca9dc9" Oct 31 00:28:23.808119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:23.807908 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e" Oct 31 00:28:25.098611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.098580 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d"] Oct 31 00:28:25.099002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.098945 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7cdf683a-1700-4276-b48d-e229a1ee79bb" containerName="curl" Oct 31 00:28:25.099002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.098957 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdf683a-1700-4276-b48d-e229a1ee79bb" containerName="curl" Oct 31 00:28:25.099089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.099015 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdf683a-1700-4276-b48d-e229a1ee79bb" containerName="curl" Oct 31 00:28:25.102073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.102055 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" Oct 31 00:28:25.108855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.108775 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"collector-namespace-auth-18e375b33f6816f7\"/\"default-dockercfg-b6ks5\"" Oct 31 00:28:25.109816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.109767 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d"] Oct 31 00:28:25.113908 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.113889 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" Oct 31 00:28:25.390269 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.390101 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d"] Oct 31 00:28:25.392850 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:25.392820 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec3a5e5d_55a8_4f31_94e5_6c98e3e9eb6b.slice/crio-3b7b1941136ed918ef0b63570a25024bcd15d357cd748536cb75c714429394bf WatchSource:0}: Error finding container 3b7b1941136ed918ef0b63570a25024bcd15d357cd748536cb75c714429394bf: Status 404 returned error can't find the container with id 3b7b1941136ed918ef0b63570a25024bcd15d357cd748536cb75c714429394bf Oct 31 00:28:25.819527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.819493 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" event={"ID":"ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b","Type":"ContainerStarted","Data":"d26507177cb73fd31ebd150e346733a392a4be7592b84516f25a74b42e0be2a9"} Oct 31 00:28:25.819801 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.819536 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" event={"ID":"ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b","Type":"ContainerStarted","Data":"3b7b1941136ed918ef0b63570a25024bcd15d357cd748536cb75c714429394bf"} Oct 31 00:28:25.838226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:25.838181 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" podStartSLOduration=0.838168211 podStartE2EDuration="838.168211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:25.83688575 +0000 UTC m=+1789.897660446" watchObservedRunningTime="2026-10-31 00:28:25.838168211 +0000 UTC m=+1789.898942906" Oct 31 00:28:33.864998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:33.864959 5073 generic.go:356] "Generic (PLEG): container finished" podID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" containerID="d26507177cb73fd31ebd150e346733a392a4be7592b84516f25a74b42e0be2a9" exitCode=0 Oct 31 00:28:33.865380 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:33.865030 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" event={"ID":"ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b","Type":"ContainerDied","Data":"d26507177cb73fd31ebd150e346733a392a4be7592b84516f25a74b42e0be2a9"} Oct 31 00:28:35.006681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.006657 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" Oct 31 00:28:35.223413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.223345 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d"] Oct 31 00:28:35.227345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.227320 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d"] Oct 31 00:28:35.248987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.248960 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e"] Oct 31 00:28:35.251279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.251253 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/enrolled-inference-allowed-18e375b5e41b635e"] Oct 31 00:28:35.273176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.273154 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593"] Oct 31 00:28:35.277719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.277691 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["collector-namespace-auth-18e375b33f6816f7/unenrolled-spoof-denied-18e375b37848c593"] Oct 31 00:28:35.877989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.877960 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" Oct 31 00:28:35.878193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:35.878001 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b7b1941136ed918ef0b63570a25024bcd15d357cd748536cb75c714429394bf" Oct 31 00:28:35.887891 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:35.887865 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"revoked-inference-denied-18e375b6e5f69c9d\" is forbidden: User \"system:node:ip-10-0-140-246.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"collector-namespace-auth-18e375b33f6816f7\": no relationship found between node 'ip-10-0-140-246.us-east-2.compute.internal' and this object" podUID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" Oct 31 00:28:36.059709 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:36.059668 5073 status_manager.go:1045] "Failed to get status for pod" err="pods \"revoked-inference-denied-18e375b6e5f69c9d\" is forbidden: User \"system:node:ip-10-0-140-246.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"collector-namespace-auth-18e375b33f6816f7\": no relationship found between node 'ip-10-0-140-246.us-east-2.compute.internal' and this object" podUID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" pod="collector-namespace-auth-18e375b33f6816f7/revoked-inference-denied-18e375b6e5f69c9d" Oct 31 00:28:36.060102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.060012 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c63f37a-52fc-4867-90b1-88605b2fd976" path="/var/lib/kubelet/pods/6c63f37a-52fc-4867-90b1-88605b2fd976/volumes" Oct 31 00:28:36.060248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.060233 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdf683a-1700-4276-b48d-e229a1ee79bb" path="/var/lib/kubelet/pods/7cdf683a-1700-4276-b48d-e229a1ee79bb/volumes" Oct 31 00:28:36.060439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.060428 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" path="/var/lib/kubelet/pods/ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b/volumes" Oct 31 00:28:36.245868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.245839 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:28:36.253093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.253072 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:28:36.253339 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.253321 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:28:36.260558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:36.260540 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:28:40.705818 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.705770 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj"] Oct 31 00:28:40.706250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.706185 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" containerName="curl" Oct 31 00:28:40.706250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.706197 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" containerName="curl" Oct 31 00:28:40.706333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.706285 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3a5e5d-55a8-4f31-94e5-6c98e3e9eb6b" containerName="curl" Oct 31 00:28:40.710837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.710817 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.720486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.720457 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj"] Oct 31 00:28:40.851103 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.851071 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.851250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.851111 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.851250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.851139 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7wm\" (UniqueName: \"kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.951618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.951590 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.951747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.951622 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.951747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.951642 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7wm\" (UniqueName: \"kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.952328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.952304 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.954215 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.954190 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:40.964052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:40.963999 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7wm\" (UniqueName: \"kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm\") pod \"data-science-collector-targetallocator-f4b97b5f4-dxxzj\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:41.025607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.025581 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:41.156161 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.156122 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db"] Oct 31 00:28:41.157527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.156392 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" podUID="6ab588f8-26c2-4321-b967-174cdf088178" containerName="thanos-querier" containerID="cri-o://1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2" gracePeriod=30 Oct 31 00:28:41.265748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.265710 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:28:41.266142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.266007 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="prometheus" containerID="cri-o://f75f91e1075ce1e4bff9b9d3506158b6ab10e015cd1e2f94a72c802634b78bae" gracePeriod=600 Oct 31 00:28:41.266142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.266039 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="thanos-sidecar" containerID="cri-o://e50a7133522a80ac87d42577498d3ca39398be0fe76ef90e6961ce3b4f22d0da" gracePeriod=600 Oct 31 00:28:41.266142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.266054 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="config-reloader" containerID="cri-o://15e32316c32417c5f992408d0f10b3ffe36e24a8d0480762ddcdc865d2405028" gracePeriod=600 Oct 31 00:28:41.335714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.335252 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj"] Oct 31 00:28:41.337489 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:41.337442 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d84c6c4_03e8_4e1d_88ff_462e5e2daf93.slice/crio-19938cc3ccbea8455568c5ef622bd714f259bd77f03f47dd07cc67ed8617da6b WatchSource:0}: Error finding container 19938cc3ccbea8455568c5ef622bd714f259bd77f03f47dd07cc67ed8617da6b: Status 404 returned error can't find the container with id 19938cc3ccbea8455568c5ef622bd714f259bd77f03f47dd07cc67ed8617da6b Oct 31 00:28:41.373486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.373445 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:28:41.373581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.373528 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:28:41.470921 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.470882 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:28:41.559152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.559082 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f\" (UniqueName: \"kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f\") pod \"6ab588f8-26c2-4321-b967-174cdf088178\" (UID: \"6ab588f8-26c2-4321-b967-174cdf088178\") " Oct 31 00:28:41.561178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.561155 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f" pod "6ab588f8-26c2-4321-b967-174cdf088178" (UID: "6ab588f8-26c2-4321-b967-174cdf088178"). InnerVolumeSpecName "kube-api-access-ngw5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.649359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.649251 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp"] Oct 31 00:28:41.649524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.649474 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" podUID="a8bd94b9-84d2-442e-9401-d45ce3d16b6b" containerName="ta-container" containerID="cri-o://727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8" gracePeriod=30 Oct 31 00:28:41.660308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.660277 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngw5f\" (UniqueName: \"kubernetes.io/projected/6ab588f8-26c2-4321-b967-174cdf088178-kube-api-access-ngw5f\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.666035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.666009 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj"] Oct 31 00:28:41.902913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.902887 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:41.913515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.913482 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" event={"ID":"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93","Type":"ContainerStarted","Data":"cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1"} Oct 31 00:28:41.913650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.913531 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:41.913650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.913547 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" event={"ID":"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93","Type":"ContainerStarted","Data":"19938cc3ccbea8455568c5ef622bd714f259bd77f03f47dd07cc67ed8617da6b"} Oct 31 00:28:41.913650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.913603 5073 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" secret="" err="secret \"data-science-collector-targetallocator-dockercfg-zpbg7\" not found" Oct 31 00:28:41.915032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.915001 5073 generic.go:356] "Generic (PLEG): container finished" podID="a8bd94b9-84d2-442e-9401-d45ce3d16b6b" containerID="727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8" exitCode=0 Oct 31 00:28:41.915122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.915060 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" Oct 31 00:28:41.915122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.915081 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" event={"ID":"a8bd94b9-84d2-442e-9401-d45ce3d16b6b","Type":"ContainerDied","Data":"727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8"} Oct 31 00:28:41.915211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.915125 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp" event={"ID":"a8bd94b9-84d2-442e-9401-d45ce3d16b6b","Type":"ContainerDied","Data":"d3b6ba61b4dd6dfa3c3d504079e8bc1c43d5b0d24c4b27f2b38f080279c0013c"} Oct 31 00:28:41.915211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.915150 5073 scope.go:122] "RemoveContainer" containerID="727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8" Oct 31 00:28:41.915561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.915541 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:41.917913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.917895 5073 generic.go:356] "Generic (PLEG): container finished" podID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerID="e50a7133522a80ac87d42577498d3ca39398be0fe76ef90e6961ce3b4f22d0da" exitCode=0 Oct 31 00:28:41.917913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.917911 5073 generic.go:356] "Generic (PLEG): container finished" podID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerID="15e32316c32417c5f992408d0f10b3ffe36e24a8d0480762ddcdc865d2405028" exitCode=0 Oct 31 00:28:41.918047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.917919 5073 generic.go:356] "Generic (PLEG): container finished" podID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerID="f75f91e1075ce1e4bff9b9d3506158b6ab10e015cd1e2f94a72c802634b78bae" exitCode=0 Oct 31 00:28:41.918047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.917968 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerDied","Data":"e50a7133522a80ac87d42577498d3ca39398be0fe76ef90e6961ce3b4f22d0da"} Oct 31 00:28:41.918047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.918030 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerDied","Data":"15e32316c32417c5f992408d0f10b3ffe36e24a8d0480762ddcdc865d2405028"} Oct 31 00:28:41.918159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.918061 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerDied","Data":"f75f91e1075ce1e4bff9b9d3506158b6ab10e015cd1e2f94a72c802634b78bae"} Oct 31 00:28:41.919320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.919298 5073 generic.go:356] "Generic (PLEG): container finished" podID="6ab588f8-26c2-4321-b967-174cdf088178" containerID="1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2" exitCode=0 Oct 31 00:28:41.919407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.919342 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" event={"ID":"6ab588f8-26c2-4321-b967-174cdf088178","Type":"ContainerDied","Data":"1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2"} Oct 31 00:28:41.919407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.919355 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" Oct 31 00:28:41.919407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.919367 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db" event={"ID":"6ab588f8-26c2-4321-b967-174cdf088178","Type":"ContainerDied","Data":"40c5e0d3453b29082f5dd433323eb0d556fbb5ce2c095379a30f7a406770cfed"} Oct 31 00:28:41.927144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.927119 5073 scope.go:122] "RemoveContainer" containerID="727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8" Oct 31 00:28:41.927708 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:41.927671 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8\": container with ID starting with 727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8 not found: ID does not exist" containerID="727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8" Oct 31 00:28:41.927844 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.927707 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8"} err="failed to get container status \"727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8\": rpc error: code = NotFound desc = could not find container \"727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8\": container with ID starting with 727c967d4bd17fbad2a3e28f904d804eeaf60177faac675b3dab1e420b0092c8 not found: ID does not exist" Oct 31 00:28:41.927844 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.927732 5073 scope.go:122] "RemoveContainer" containerID="1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2" Oct 31 00:28:41.938980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.938963 5073 scope.go:122] "RemoveContainer" containerID="1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2" Oct 31 00:28:41.939257 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:41.939236 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2\": container with ID starting with 1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2 not found: ID does not exist" containerID="1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2" Oct 31 00:28:41.939329 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.939263 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2"} err="failed to get container status \"1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2\": rpc error: code = NotFound desc = could not find container \"1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2\": container with ID starting with 1009b10e538bc5ea3d21198d63dc73fd583bc39b0c08556eba7cef021070efd2 not found: ID does not exist" Oct 31 00:28:41.949064 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.949023 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" podStartSLOduration=1.9490114570000001 podStartE2EDuration="1.949011457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:41.947743938 +0000 UTC m=+1806.008518672" watchObservedRunningTime="2026-10-31 00:28:41.949011457 +0000 UTC m=+1806.009786152" Oct 31 00:28:41.966746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.966727 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db"] Oct 31 00:28:41.973249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:41.973229 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tw7db"] Oct 31 00:28:42.059147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.059112 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab588f8-26c2-4321-b967-174cdf088178" path="/var/lib/kubelet/pods/6ab588f8-26c2-4321-b967-174cdf088178/volumes" Oct 31 00:28:42.063829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.063805 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert\") pod \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " Oct 31 00:28:42.063954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.063838 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt\" (UniqueName: \"kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt\") pod \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " Oct 31 00:28:42.063954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.063889 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal\" (UniqueName: \"kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal\") pod \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\" (UID: \"a8bd94b9-84d2-442e-9401-d45ce3d16b6b\") " Oct 31 00:28:42.064388 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:42.064150 5073 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-targetallocator: configmap "data-science-collector-targetallocator" not found Oct 31 00:28:42.064388 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:42.064220 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal podName:2d84c6c4-03e8-4e1d-88ff-462e5e2daf93 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:42.56419977 +0000 UTC m=+1806.624974449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ta-internal" (UniqueName: "kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal") pod "data-science-collector-targetallocator-f4b97b5f4-dxxzj" (UID: "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93") : configmap "data-science-collector-targetallocator" not found Oct 31 00:28:42.064388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.064337 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal" pod "a8bd94b9-84d2-442e-9401-d45ce3d16b6b" (UID: "a8bd94b9-84d2-442e-9401-d45ce3d16b6b"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.066389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.066365 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert" pod "a8bd94b9-84d2-442e-9401-d45ce3d16b6b" (UID: "a8bd94b9-84d2-442e-9401-d45ce3d16b6b"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.066541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.066527 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt" pod "a8bd94b9-84d2-442e-9401-d45ce3d16b6b" (UID: "a8bd94b9-84d2-442e-9401-d45ce3d16b6b"). InnerVolumeSpecName "kube-api-access-z8ngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.118720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.118701 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:28:42.165530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.165508 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.165530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.165531 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8ngt\" (UniqueName: \"kubernetes.io/projected/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-kube-api-access-z8ngt\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.165676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.165548 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/a8bd94b9-84d2-442e-9401-d45ce3d16b6b-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.238295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.238263 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp"] Oct 31 00:28:42.244286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.244258 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-77f8cbbcbf-kchmp"] Oct 31 00:28:42.266670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266641 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.266771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266699 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.266771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266725 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.266894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266843 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.266894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266865 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.266894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266887 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.266954 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267021 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267023 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.267281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267066 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267093 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267141 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267167 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config\") pod \"d311ed02-63b1-4c82-9c7c-019c23f82869\" (UID: \"d311ed02-63b1-4c82-9c7c-019c23f82869\") " Oct 31 00:28:42.267281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267238 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.267696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267477 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.267696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.267499 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.268173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.268140 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.269969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.269926 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.270073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.269983 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.270152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.270091 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:42.270538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.270500 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "kube-api-access-tjjzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.270738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.270718 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.271013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.270932 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.271674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.271653 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.272004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.271983 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.273304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.273283 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:28:42.283360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.283340 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config" pod "d311ed02-63b1-4c82-9c7c-019c23f82869" (UID: "d311ed02-63b1-4c82-9c7c-019c23f82869"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.368292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368242 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368264 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368274 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368283 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjzp\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-kube-api-access-tjjzp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368292 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d311ed02-63b1-4c82-9c7c-019c23f82869-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368301 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368312 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d311ed02-63b1-4c82-9c7c-019c23f82869-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368334 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:28:42.368515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368344 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368353 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d311ed02-63b1-4c82-9c7c-019c23f82869-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.368515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.368361 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d311ed02-63b1-4c82-9c7c-019c23f82869-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.379941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.379926 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:28:42.468818 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.468773 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" Oct 31 00:28:42.569335 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:42.569315 5073 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-targetallocator: configmap "data-science-collector-targetallocator" not found Oct 31 00:28:42.569457 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:42.569374 5073 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal podName:2d84c6c4-03e8-4e1d-88ff-462e5e2daf93 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.569361095 +0000 UTC m=+1807.630135768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ta-internal" (UniqueName: "kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal") pod "data-science-collector-targetallocator-f4b97b5f4-dxxzj" (UID: "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93") : configmap "data-science-collector-targetallocator" not found Oct 31 00:28:42.928826 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.928767 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d311ed02-63b1-4c82-9c7c-019c23f82869","Type":"ContainerDied","Data":"3b4fb45c758a4a57510eb8b3bb29151b39eb056331f70c0235c4d706f051fe2f"} Oct 31 00:28:42.928826 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.928824 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:28:42.929323 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.928886 5073 scope.go:122] "RemoveContainer" containerID="e50a7133522a80ac87d42577498d3ca39398be0fe76ef90e6961ce3b4f22d0da" Oct 31 00:28:42.942592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.942556 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" podUID="2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" containerName="ta-container" containerID="cri-o://cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1" gracePeriod=30 Oct 31 00:28:42.943820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.943801 5073 scope.go:122] "RemoveContainer" containerID="15e32316c32417c5f992408d0f10b3ffe36e24a8d0480762ddcdc865d2405028" Oct 31 00:28:42.960562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.960544 5073 scope.go:122] "RemoveContainer" containerID="f75f91e1075ce1e4bff9b9d3506158b6ab10e015cd1e2f94a72c802634b78bae" Oct 31 00:28:42.980738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.980713 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:28:42.987003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:42.986982 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:28:43.036031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.036011 5073 scope.go:122] "RemoveContainer" containerID="159e2fc044776298155eadf272ddf847d645ac2961c877fa371735d66425cf15" Oct 31 00:28:43.085453 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:43.085419 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:28:43.185681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.185632 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:43.276870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.276835 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert\") pod \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " Oct 31 00:28:43.277056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.276879 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal\" (UniqueName: \"kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal\") pod \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " Oct 31 00:28:43.277056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.276910 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm\" (UniqueName: \"kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm\") pod \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\" (UID: \"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93\") " Oct 31 00:28:43.277318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.277290 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal" pod "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" (UID: "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:43.279220 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.279198 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert" pod "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" (UID: "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:43.279285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.279251 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm" pod "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" (UID: "2d84c6c4-03e8-4e1d-88ff-462e5e2daf93"). InnerVolumeSpecName "kube-api-access-vh7wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:43.377935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.377910 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:43.377935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.377935 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:43.378072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.377950 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7wm\" (UniqueName: \"kubernetes.io/projected/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93-kube-api-access-vh7wm\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:43.951734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.951702 5073 generic.go:356] "Generic (PLEG): container finished" podID="2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" containerID="cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1" exitCode=0 Oct 31 00:28:43.952257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.951758 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" Oct 31 00:28:43.952257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.951762 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" event={"ID":"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93","Type":"ContainerDied","Data":"cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1"} Oct 31 00:28:43.952257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.951883 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj" event={"ID":"2d84c6c4-03e8-4e1d-88ff-462e5e2daf93","Type":"ContainerDied","Data":"19938cc3ccbea8455568c5ef622bd714f259bd77f03f47dd07cc67ed8617da6b"} Oct 31 00:28:43.952257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.951908 5073 scope.go:122] "RemoveContainer" containerID="cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1" Oct 31 00:28:43.966124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.966108 5073 scope.go:122] "RemoveContainer" containerID="cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1" Oct 31 00:28:43.966372 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:28:43.966350 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1\": container with ID starting with cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1 not found: ID does not exist" containerID="cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1" Oct 31 00:28:43.966426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.966392 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1"} err="failed to get container status \"cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1\": rpc error: code = NotFound desc = could not find container \"cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1\": container with ID starting with cfbc8072f9af7a7769cdff7cd3aebca3e4e5ae2e01481f605a8bbe499a7fd4c1 not found: ID does not exist" Oct 31 00:28:43.976639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.976612 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj"] Oct 31 00:28:43.981459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:43.981436 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f4b97b5f4-dxxzj"] Oct 31 00:28:44.058252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.058217 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" path="/var/lib/kubelet/pods/2d84c6c4-03e8-4e1d-88ff-462e5e2daf93/volumes" Oct 31 00:28:44.058609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.058594 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bd94b9-84d2-442e-9401-d45ce3d16b6b" path="/var/lib/kubelet/pods/a8bd94b9-84d2-442e-9401-d45ce3d16b6b/volumes" Oct 31 00:28:44.059018 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.059005 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" path="/var/lib/kubelet/pods/d311ed02-63b1-4c82-9c7c-019c23f82869/volumes" Oct 31 00:28:44.859953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.859919 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:28:44.860277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860250 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="init-config-reloader" Oct 31 00:28:44.860277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860267 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="init-config-reloader" Oct 31 00:28:44.860277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860277 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="config-reloader" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860281 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="config-reloader" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860302 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a8bd94b9-84d2-442e-9401-d45ce3d16b6b" containerName="ta-container" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860308 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bd94b9-84d2-442e-9401-d45ce3d16b6b" containerName="ta-container" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860320 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6ab588f8-26c2-4321-b967-174cdf088178" containerName="thanos-querier" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860327 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab588f8-26c2-4321-b967-174cdf088178" containerName="thanos-querier" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860340 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="prometheus" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860344 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="prometheus" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860350 5073 cpu_manager.go:414] "containerMap: removing container" podUID="2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" containerName="ta-container" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860354 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" containerName="ta-container" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860364 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="thanos-sidecar" Oct 31 00:28:44.860396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860368 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="thanos-sidecar" Oct 31 00:28:44.860727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860429 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab588f8-26c2-4321-b967-174cdf088178" containerName="thanos-querier" Oct 31 00:28:44.860727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860436 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="prometheus" Oct 31 00:28:44.860727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860442 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d84c6c4-03e8-4e1d-88ff-462e5e2daf93" containerName="ta-container" Oct 31 00:28:44.860727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860448 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="thanos-sidecar" Oct 31 00:28:44.860727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860454 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d311ed02-63b1-4c82-9c7c-019c23f82869" containerName="config-reloader" Oct 31 00:28:44.860727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.860461 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bd94b9-84d2-442e-9401-d45ce3d16b6b" containerName="ta-container" Oct 31 00:28:44.865232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.865161 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:44.880658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.880623 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:28:44.965275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.965246 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:28:44.965275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.965271 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:28:44.965643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.965342 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:28:44.966699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.966676 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lx78f\"" Oct 31 00:28:44.966820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.966735 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:28:44.966866 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.966824 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:28:44.990164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.990140 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878pq\" (UniqueName: \"kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:44.990254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.990193 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:44.990311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.990259 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:44.990311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:44.990289 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.090945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.090920 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.091043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.090957 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.091043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.090973 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.091043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.091014 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878pq\" (UniqueName: \"kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.091711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.091645 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.093423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.093397 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.093573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.093558 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.104202 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.104183 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-878pq\" (UniqueName: \"kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq\") pod \"data-science-collector-collector-1\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.178678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.178632 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:45.456191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.456121 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:28:45.459240 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:28:45.459210 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d163056_6343_494f_9e50_b1cc278bec84.slice/crio-d8a9cfbe6cca8921419d84c0ca0bb4541c8158d3e0382320f5f10cc241368d46 WatchSource:0}: Error finding container d8a9cfbe6cca8921419d84c0ca0bb4541c8158d3e0382320f5f10cc241368d46: Status 404 returned error can't find the container with id d8a9cfbe6cca8921419d84c0ca0bb4541c8158d3e0382320f5f10cc241368d46 Oct 31 00:28:45.461284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.461254 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:28:45.461335 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.461313 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:28:45.966252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.966207 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d163056-6343-494f-9e50-b1cc278bec84","Type":"ContainerStarted","Data":"013af769948cea1beda1c0c0238bbace55036c606e04920b5a6eb3ef2d115429"} Oct 31 00:28:45.966642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.966270 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d163056-6343-494f-9e50-b1cc278bec84","Type":"ContainerStarted","Data":"d8a9cfbe6cca8921419d84c0ca0bb4541c8158d3e0382320f5f10cc241368d46"} Oct 31 00:28:45.986636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:28:45.986581 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.986564763 podStartE2EDuration="1.986564763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:45.985294684 +0000 UTC m=+1810.046069391" watchObservedRunningTime="2026-10-31 00:28:45.986564763 +0000 UTC m=+1810.047339458" Oct 31 00:29:13.084526 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:29:13.084492 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:29:35.109534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.109502 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:35.110097 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.109752 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7d163056-6343-494f-9e50-b1cc278bec84" containerName="otc-container" containerID="cri-o://013af769948cea1beda1c0c0238bbace55036c606e04920b5a6eb3ef2d115429" gracePeriod=30 Oct 31 00:29:35.235188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.235155 5073 generic.go:356] "Generic (PLEG): container finished" podID="7d163056-6343-494f-9e50-b1cc278bec84" containerID="013af769948cea1beda1c0c0238bbace55036c606e04920b5a6eb3ef2d115429" exitCode=0 Oct 31 00:29:35.235328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.235209 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d163056-6343-494f-9e50-b1cc278bec84","Type":"ContainerDied","Data":"013af769948cea1beda1c0c0238bbace55036c606e04920b5a6eb3ef2d115429"} Oct 31 00:29:35.377543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.377520 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:35.460760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.460727 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal\") pod \"7d163056-6343-494f-9e50-b1cc278bec84\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " Oct 31 00:29:35.460937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.460775 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs\") pod \"7d163056-6343-494f-9e50-b1cc278bec84\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " Oct 31 00:29:35.460937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.460858 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs\") pod \"7d163056-6343-494f-9e50-b1cc278bec84\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " Oct 31 00:29:35.461123 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.460969 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq\" (UniqueName: \"kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq\") pod \"7d163056-6343-494f-9e50-b1cc278bec84\" (UID: \"7d163056-6343-494f-9e50-b1cc278bec84\") " Oct 31 00:29:35.461165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.461145 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal" pod "7d163056-6343-494f-9e50-b1cc278bec84" (UID: "7d163056-6343-494f-9e50-b1cc278bec84"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:35.461333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.461317 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d163056-6343-494f-9e50-b1cc278bec84-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.463085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.463060 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs" pod "7d163056-6343-494f-9e50-b1cc278bec84" (UID: "7d163056-6343-494f-9e50-b1cc278bec84"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:35.463085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.463076 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs" pod "7d163056-6343-494f-9e50-b1cc278bec84" (UID: "7d163056-6343-494f-9e50-b1cc278bec84"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:35.463245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.463193 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq" pod "7d163056-6343-494f-9e50-b1cc278bec84" (UID: "7d163056-6343-494f-9e50-b1cc278bec84"). InnerVolumeSpecName "kube-api-access-878pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:35.561792 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.561765 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.561905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.561805 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d163056-6343-494f-9e50-b1cc278bec84-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.561905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:35.561816 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878pq\" (UniqueName: \"kubernetes.io/projected/7d163056-6343-494f-9e50-b1cc278bec84-kube-api-access-878pq\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.241725 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:36.241693 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7d163056-6343-494f-9e50-b1cc278bec84","Type":"ContainerDied","Data":"d8a9cfbe6cca8921419d84c0ca0bb4541c8158d3e0382320f5f10cc241368d46"} Oct 31 00:29:36.242211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:36.241728 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:36.242211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:36.241750 5073 scope.go:122] "RemoveContainer" containerID="013af769948cea1beda1c0c0238bbace55036c606e04920b5a6eb3ef2d115429" Oct 31 00:29:36.260517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:36.260494 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:36.271300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:36.271268 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:38.062896 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.062857 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d163056-6343-494f-9e50-b1cc278bec84" path="/var/lib/kubelet/pods/7d163056-6343-494f-9e50-b1cc278bec84/volumes" Oct 31 00:29:38.181626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.181545 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:38.182568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.182543 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7d163056-6343-494f-9e50-b1cc278bec84" containerName="otc-container" Oct 31 00:29:38.182568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.182568 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d163056-6343-494f-9e50-b1cc278bec84" containerName="otc-container" Oct 31 00:29:38.182751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.182647 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d163056-6343-494f-9e50-b1cc278bec84" containerName="otc-container" Oct 31 00:29:38.187590 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.187567 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.192499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.192468 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:29:38.192499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.192477 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:29:38.193712 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.193692 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:38.193843 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.193764 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:38.193922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.193890 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:29:38.203513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.203491 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:38.267004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.266973 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m629n\"" Oct 31 00:29:38.282676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.282646 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.282842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.282684 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.282842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.282722 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.282842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.282779 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxhv\" (UniqueName: \"kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.383181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.383113 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.383181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.383163 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.383181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.383179 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxhv\" (UniqueName: \"kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.383388 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.383224 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.383762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.383735 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.385966 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.385935 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.386066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.385961 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.396052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.396030 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxhv\" (UniqueName: \"kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv\") pod \"data-science-collector-collector-1\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.506831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.506794 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:38.790433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.790408 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:38.793183 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:29:38.793122 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2f080d_ffda_49b8_abee_b1d2b943c3a7.slice/crio-13ebf30f9557048ea7dd6bd8c3fba5efe1816fcf635a6e539d83f3986da98b03 WatchSource:0}: Error finding container 13ebf30f9557048ea7dd6bd8c3fba5efe1816fcf635a6e539d83f3986da98b03: Status 404 returned error can't find the container with id 13ebf30f9557048ea7dd6bd8c3fba5efe1816fcf635a6e539d83f3986da98b03 Oct 31 00:29:38.795367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.795328 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:29:38.795431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:38.795395 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:29:39.259395 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:39.259356 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df2f080d-ffda-49b8-abee-b1d2b943c3a7","Type":"ContainerStarted","Data":"8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580"} Oct 31 00:29:39.259771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:39.259423 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df2f080d-ffda-49b8-abee-b1d2b943c3a7","Type":"ContainerStarted","Data":"13ebf30f9557048ea7dd6bd8c3fba5efe1816fcf635a6e539d83f3986da98b03"} Oct 31 00:29:39.281524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:39.281480 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.281466693 podStartE2EDuration="1.281466693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:39.279449702 +0000 UTC m=+1863.340224399" watchObservedRunningTime="2026-10-31 00:29:39.281466693 +0000 UTC m=+1863.342241388" Oct 31 00:29:43.085409 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:29:43.085375 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:29:52.026141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.026105 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:52.026659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.026323 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="df2f080d-ffda-49b8-abee-b1d2b943c3a7" containerName="otc-container" containerID="cri-o://8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580" gracePeriod=30 Oct 31 00:29:52.058155 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:29:52.058129 5073 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2f080d_ffda_49b8_abee_b1d2b943c3a7.slice/crio-conmon-8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:29:52.309487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.309463 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:52.328665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.328347 5073 generic.go:356] "Generic (PLEG): container finished" podID="df2f080d-ffda-49b8-abee-b1d2b943c3a7" containerID="8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580" exitCode=0 Oct 31 00:29:52.328665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.328539 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df2f080d-ffda-49b8-abee-b1d2b943c3a7","Type":"ContainerDied","Data":"8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580"} Oct 31 00:29:52.328665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.328587 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:52.329054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.328594 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df2f080d-ffda-49b8-abee-b1d2b943c3a7","Type":"ContainerDied","Data":"13ebf30f9557048ea7dd6bd8c3fba5efe1816fcf635a6e539d83f3986da98b03"} Oct 31 00:29:52.329054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.328608 5073 scope.go:122] "RemoveContainer" containerID="8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580" Oct 31 00:29:52.338687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.338671 5073 scope.go:122] "RemoveContainer" containerID="8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580" Oct 31 00:29:52.339039 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:29:52.339012 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580\": container with ID starting with 8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580 not found: ID does not exist" containerID="8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580" Oct 31 00:29:52.339123 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.339052 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580"} err="failed to get container status \"8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580\": rpc error: code = NotFound desc = could not find container \"8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580\": container with ID starting with 8710914f8b3958ed343844dbf75b9f43d2924d31d2550842537558a60731d580 not found: ID does not exist" Oct 31 00:29:52.395297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.395270 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs\") pod \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " Oct 31 00:29:52.395407 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.395312 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal\") pod \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " Oct 31 00:29:52.395480 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.395441 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv\" (UniqueName: \"kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv\") pod \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " Oct 31 00:29:52.395548 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.395530 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs\") pod \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\" (UID: \"df2f080d-ffda-49b8-abee-b1d2b943c3a7\") " Oct 31 00:29:52.395633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.395615 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal" pod "df2f080d-ffda-49b8-abee-b1d2b943c3a7" (UID: "df2f080d-ffda-49b8-abee-b1d2b943c3a7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:52.395832 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.395806 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.397512 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.397486 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv" pod "df2f080d-ffda-49b8-abee-b1d2b943c3a7" (UID: "df2f080d-ffda-49b8-abee-b1d2b943c3a7"). InnerVolumeSpecName "kube-api-access-nhxhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:52.397632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.397610 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs" pod "df2f080d-ffda-49b8-abee-b1d2b943c3a7" (UID: "df2f080d-ffda-49b8-abee-b1d2b943c3a7"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:52.397677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.397625 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs" pod "df2f080d-ffda-49b8-abee-b1d2b943c3a7" (UID: "df2f080d-ffda-49b8-abee-b1d2b943c3a7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:52.496817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.496766 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.496817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.496814 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f080d-ffda-49b8-abee-b1d2b943c3a7-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.496817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.496826 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxhv\" (UniqueName: \"kubernetes.io/projected/df2f080d-ffda-49b8-abee-b1d2b943c3a7-kube-api-access-nhxhv\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.649687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.649660 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:52.654838 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:52.654816 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:54.058195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.058162 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2f080d-ffda-49b8-abee-b1d2b943c3a7" path="/var/lib/kubelet/pods/df2f080d-ffda-49b8-abee-b1d2b943c3a7/volumes" Oct 31 00:29:54.979330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.979292 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:54.979871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.979851 5073 cpu_manager.go:414] "containerMap: removing container" podUID="df2f080d-ffda-49b8-abee-b1d2b943c3a7" containerName="otc-container" Oct 31 00:29:54.979953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.979873 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2f080d-ffda-49b8-abee-b1d2b943c3a7" containerName="otc-container" Oct 31 00:29:54.980001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.979987 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2f080d-ffda-49b8-abee-b1d2b943c3a7" containerName="otc-container" Oct 31 00:29:54.984668 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.984643 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:54.991886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.991839 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:29:54.992059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.991951 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-ck9kg\"" Oct 31 00:29:54.992059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.992033 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:29:54.993565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:54.993484 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:55.063080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.063057 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:55.063418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.063216 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:55.063418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.063242 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:29:55.117652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.117630 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.117867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.117669 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.117867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.117688 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.117867 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.117752 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcgq\" (UniqueName: \"kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.218833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.218808 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.218949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.218935 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.219013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.218997 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcgq\" (UniqueName: \"kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.219369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.219344 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.219727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.219706 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.221500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.221483 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.221810 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.221773 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.231624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.231606 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcgq\" (UniqueName: \"kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq\") pod \"data-science-collector-collector-1\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.300493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.300469 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:29:55.578467 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.578383 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:29:55.582150 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:29:55.582122 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45004def_2cc9_40c9_aa04_752076dbd542.slice/crio-b9ca11bfe35ff025a59f05bbf45c5e10b368b8e5a1b81be8b2004ee989aa7f73 WatchSource:0}: Error finding container b9ca11bfe35ff025a59f05bbf45c5e10b368b8e5a1b81be8b2004ee989aa7f73: Status 404 returned error can't find the container with id b9ca11bfe35ff025a59f05bbf45c5e10b368b8e5a1b81be8b2004ee989aa7f73 Oct 31 00:29:55.584225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.584193 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:29:55.584320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:55.584261 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:29:56.353358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:56.353318 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"45004def-2cc9-40c9-aa04-752076dbd542","Type":"ContainerStarted","Data":"9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c"} Oct 31 00:29:56.353834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:56.353375 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"45004def-2cc9-40c9-aa04-752076dbd542","Type":"ContainerStarted","Data":"b9ca11bfe35ff025a59f05bbf45c5e10b368b8e5a1b81be8b2004ee989aa7f73"} Oct 31 00:29:56.376538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:29:56.376491 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.376478255 podStartE2EDuration="2.376478255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:56.374521225 +0000 UTC m=+1880.435295955" watchObservedRunningTime="2026-10-31 00:29:56.376478255 +0000 UTC m=+1880.437252950" Oct 31 00:30:13.086849 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:30:13.086812 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:30:36.590206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:36.590170 5073 scope.go:122] "RemoveContainer" containerID="8b5b08bebce0bd2535d806a60f589d14fa69c34f8998e759e2c5b685a35bc3c1" Oct 31 00:30:43.086702 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:30:43.086666 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:30:47.270691 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.270615 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:30:47.271859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.271270 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="45004def-2cc9-40c9-aa04-752076dbd542" containerName="otc-container" containerID="cri-o://9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c" gracePeriod=30 Oct 31 00:30:47.551098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.551066 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:30:47.608775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.608742 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs\") pod \"45004def-2cc9-40c9-aa04-752076dbd542\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " Oct 31 00:30:47.608961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.608778 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs\") pod \"45004def-2cc9-40c9-aa04-752076dbd542\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " Oct 31 00:30:47.608961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.608873 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq\" (UniqueName: \"kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq\") pod \"45004def-2cc9-40c9-aa04-752076dbd542\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " Oct 31 00:30:47.608961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.608919 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal\" (UniqueName: \"kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal\") pod \"45004def-2cc9-40c9-aa04-752076dbd542\" (UID: \"45004def-2cc9-40c9-aa04-752076dbd542\") " Oct 31 00:30:47.609399 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.609369 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal" pod "45004def-2cc9-40c9-aa04-752076dbd542" (UID: "45004def-2cc9-40c9-aa04-752076dbd542"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:47.611041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.611012 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs" pod "45004def-2cc9-40c9-aa04-752076dbd542" (UID: "45004def-2cc9-40c9-aa04-752076dbd542"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:47.611141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.611052 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq" pod "45004def-2cc9-40c9-aa04-752076dbd542" (UID: "45004def-2cc9-40c9-aa04-752076dbd542"). InnerVolumeSpecName "kube-api-access-xmcgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:47.611141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.611113 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs" pod "45004def-2cc9-40c9-aa04-752076dbd542" (UID: "45004def-2cc9-40c9-aa04-752076dbd542"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:47.626755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.626732 5073 generic.go:356] "Generic (PLEG): container finished" podID="45004def-2cc9-40c9-aa04-752076dbd542" containerID="9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c" exitCode=0 Oct 31 00:30:47.626872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.626779 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:30:47.626872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.626814 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"45004def-2cc9-40c9-aa04-752076dbd542","Type":"ContainerDied","Data":"9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c"} Oct 31 00:30:47.626954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.626893 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"45004def-2cc9-40c9-aa04-752076dbd542","Type":"ContainerDied","Data":"b9ca11bfe35ff025a59f05bbf45c5e10b368b8e5a1b81be8b2004ee989aa7f73"} Oct 31 00:30:47.626954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.626916 5073 scope.go:122] "RemoveContainer" containerID="9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c" Oct 31 00:30:47.636578 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.636563 5073 scope.go:122] "RemoveContainer" containerID="9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c" Oct 31 00:30:47.636867 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:30:47.636842 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c\": container with ID starting with 9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c not found: ID does not exist" containerID="9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c" Oct 31 00:30:47.636960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.636878 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c"} err="failed to get container status \"9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c\": rpc error: code = NotFound desc = could not find container \"9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c\": container with ID starting with 9d768b5351f8509ed044f6900f28af09ddfb29134fcc043c522334060bc4258c not found: ID does not exist" Oct 31 00:30:47.652008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.651985 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:30:47.654249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.654219 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:30:47.709511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.709481 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.709511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.709511 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/45004def-2cc9-40c9-aa04-752076dbd542-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.709675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.709526 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmcgq\" (UniqueName: \"kubernetes.io/projected/45004def-2cc9-40c9-aa04-752076dbd542-kube-api-access-xmcgq\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.709675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:47.709540 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/45004def-2cc9-40c9-aa04-752076dbd542-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.059050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:30:48.059008 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45004def-2cc9-40c9-aa04-752076dbd542" path="/var/lib/kubelet/pods/45004def-2cc9-40c9-aa04-752076dbd542/volumes" Oct 31 00:31:13.085914 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:31:13.085877 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:31:13.907729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.907700 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:31:13.908036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.908024 5073 cpu_manager.go:414] "containerMap: removing container" podUID="45004def-2cc9-40c9-aa04-752076dbd542" containerName="otc-container" Oct 31 00:31:13.908095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.908038 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45004def-2cc9-40c9-aa04-752076dbd542" containerName="otc-container" Oct 31 00:31:13.908134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.908112 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="45004def-2cc9-40c9-aa04-752076dbd542" containerName="otc-container" Oct 31 00:31:13.910983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.910964 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:13.919195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.919169 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:31:13.919345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.919251 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:31:13.919345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.919280 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:31:13.920160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:13.920140 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:31:14.012006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.011972 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjbp\" (UniqueName: \"kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp\") pod \"seaweedfs-bucket-creator\" (UID: \"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:14.112960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.112930 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjbp\" (UniqueName: \"kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp\") pod \"seaweedfs-bucket-creator\" (UID: \"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:14.125900 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.125873 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjbp\" (UniqueName: \"kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp\") pod \"seaweedfs-bucket-creator\" (UID: \"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:14.231195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.231178 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:14.502337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.502261 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:31:14.505375 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:31:14.505347 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea03fe6_c2c6_4a2e_a9fd_10e101e68d99.slice/crio-d3d4b7b22b8d944d7f57f22cb401680bffc2d896a8abc5a3103aeb7fb16e7bfe WatchSource:0}: Error finding container d3d4b7b22b8d944d7f57f22cb401680bffc2d896a8abc5a3103aeb7fb16e7bfe: Status 404 returned error can't find the container with id d3d4b7b22b8d944d7f57f22cb401680bffc2d896a8abc5a3103aeb7fb16e7bfe Oct 31 00:31:14.507240 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.507220 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:31:14.771432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:14.771355 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99","Type":"ContainerStarted","Data":"d3d4b7b22b8d944d7f57f22cb401680bffc2d896a8abc5a3103aeb7fb16e7bfe"} Oct 31 00:31:24.834233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:24.834201 5073 generic.go:356] "Generic (PLEG): container finished" podID="eea03fe6-c2c6-4a2e-a9fd-10e101e68d99" containerID="37631a4b5e5987a521719039c75c64dc7f31ee726dd94b2e86f448185309f855" exitCode=0 Oct 31 00:31:24.834647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:24.834338 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99","Type":"ContainerDied","Data":"37631a4b5e5987a521719039c75c64dc7f31ee726dd94b2e86f448185309f855"} Oct 31 00:31:25.961427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:25.961403 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:26.010903 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:26.010877 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp\" (UniqueName: \"kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp\") pod \"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99\" (UID: \"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99\") " Oct 31 00:31:26.013000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:26.012972 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp" pod "eea03fe6-c2c6-4a2e-a9fd-10e101e68d99" (UID: "eea03fe6-c2c6-4a2e-a9fd-10e101e68d99"). InnerVolumeSpecName "kube-api-access-ctjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:26.112720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:26.112646 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctjbp\" (UniqueName: \"kubernetes.io/projected/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99-kube-api-access-ctjbp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:26.846917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:26.846886 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:31:26.846917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:26.846911 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"eea03fe6-c2c6-4a2e-a9fd-10e101e68d99","Type":"ContainerDied","Data":"d3d4b7b22b8d944d7f57f22cb401680bffc2d896a8abc5a3103aeb7fb16e7bfe"} Oct 31 00:31:26.847132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:26.846946 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3d4b7b22b8d944d7f57f22cb401680bffc2d896a8abc5a3103aeb7fb16e7bfe" Oct 31 00:31:27.115885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.115806 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:31:27.116352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.116158 5073 cpu_manager.go:414] "containerMap: removing container" podUID="eea03fe6-c2c6-4a2e-a9fd-10e101e68d99" containerName="create-bucket" Oct 31 00:31:27.116352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.116170 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea03fe6-c2c6-4a2e-a9fd-10e101e68d99" containerName="create-bucket" Oct 31 00:31:27.116352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.116248 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea03fe6-c2c6-4a2e-a9fd-10e101e68d99" containerName="create-bucket" Oct 31 00:31:27.119693 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.119651 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.130802 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.130765 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:31:27.130930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.130853 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:31:27.132210 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.132189 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:31:27.132609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.132584 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rkjdd\"" Oct 31 00:31:27.132706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.132684 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:31:27.132774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.132750 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:31:27.134644 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.134619 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:31:27.220935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.220904 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.221066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.220954 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.221066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.221044 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.221178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.221082 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fhf\" (UniqueName: \"kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.322447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.322418 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.322626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.322459 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.322626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.322482 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.322626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.322506 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fhf\" (UniqueName: \"kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.323212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.323180 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.325067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.325044 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.325230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.325208 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.334914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.334893 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fhf\" (UniqueName: \"kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf\") pod \"data-science-collector-collector-0\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.434726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.434676 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:31:27.710167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.709980 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:31:27.712755 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:31:27.712727 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d439e5_d488_4132_a59f_85302ba62a44.slice/crio-eda55b92613df9760520591c4c2e84b9d7810228360ff29dcda75a3b9b85b05a WatchSource:0}: Error finding container eda55b92613df9760520591c4c2e84b9d7810228360ff29dcda75a3b9b85b05a: Status 404 returned error can't find the container with id eda55b92613df9760520591c4c2e84b9d7810228360ff29dcda75a3b9b85b05a Oct 31 00:31:27.714981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.714948 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:31:27.715048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.715015 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:31:27.854718 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.854682 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49d439e5-d488-4132-a59f-85302ba62a44","Type":"ContainerStarted","Data":"a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2"} Oct 31 00:31:27.854933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.854737 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49d439e5-d488-4132-a59f-85302ba62a44","Type":"ContainerStarted","Data":"eda55b92613df9760520591c4c2e84b9d7810228360ff29dcda75a3b9b85b05a"} Oct 31 00:31:27.873811 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:27.873748 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.873734918 podStartE2EDuration="873.734918ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:27.872818944 +0000 UTC m=+1971.933593640" watchObservedRunningTime="2026-10-31 00:31:27.873734918 +0000 UTC m=+1971.934509612" Oct 31 00:31:30.438726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.438691 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp"] Oct 31 00:31:30.442386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.442370 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.447997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.447964 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 31 00:31:30.447997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.447972 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 31 00:31:30.447997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.447988 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:31:30.448198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.447981 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 31 00:31:30.448198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.447965 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:31:30.448198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.447980 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 31 00:31:30.448198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.448036 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-h9nc8\"" Oct 31 00:31:30.455993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.455972 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp"] Oct 31 00:31:30.547667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547638 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggs7\" (UniqueName: \"kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.547838 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547690 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.547838 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547805 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.547967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547839 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.547967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547921 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.547967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547952 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.548107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.547979 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648435 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648473 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648492 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648531 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggs7\" (UniqueName: \"kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648574 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648604 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.648918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.648632 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.649212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.649191 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.649424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.649321 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.649555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.649515 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.651276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.651256 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.651367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.651278 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.651410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.651370 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.660776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.660756 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggs7\" (UniqueName: \"kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7\") pod \"tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:30.758204 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:30.758176 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:31.039488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:31.039396 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp"] Oct 31 00:31:31.042350 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:31:31.042321 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16840ea_0cbd_424f_8d07_f7f57eaa3d3e.slice/crio-fb42911e201429cf3a3a98b7def23a8078ee7b2c93689626cccf0761ae0747c0 WatchSource:0}: Error finding container fb42911e201429cf3a3a98b7def23a8078ee7b2c93689626cccf0761ae0747c0: Status 404 returned error can't find the container with id fb42911e201429cf3a3a98b7def23a8078ee7b2c93689626cccf0761ae0747c0 Oct 31 00:31:31.878087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:31.878053 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" event={"ID":"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e","Type":"ContainerStarted","Data":"fb42911e201429cf3a3a98b7def23a8078ee7b2c93689626cccf0761ae0747c0"} Oct 31 00:31:36.615349 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:36.615310 5073 scope.go:122] "RemoveContainer" containerID="8e8122892897d14ecd40f5f84e36ddc31397d41a1f1dab81a30508328c01784d" Oct 31 00:31:36.626976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:36.626957 5073 scope.go:122] "RemoveContainer" containerID="7d4c0c198aefc4d0722278c1030bc3a178a914ac0707e2f62c1de4c4b49bc456" Oct 31 00:31:36.636040 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:36.636017 5073 scope.go:122] "RemoveContainer" containerID="cf856148894c62c19c5cdcd31371df47e3723d62320700eba80d376667d35a61" Oct 31 00:31:36.645319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:36.645296 5073 scope.go:122] "RemoveContainer" containerID="8ed691c21556a1e3c9be4e761e42e7ec10ac49a9652d3b7f5bf489dd8a210ab2" Oct 31 00:31:36.654068 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:36.654041 5073 scope.go:122] "RemoveContainer" containerID="3aa0264bd961e1e3dec3c22b0acfadbc330fe53dd8799129711c2650c0825d3f" Oct 31 00:31:36.663657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:36.663631 5073 scope.go:122] "RemoveContainer" containerID="04198ff38e965e80dde161d80107c0531064422d765b8ba73236f4edb20855da" Oct 31 00:31:37.913583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:37.913549 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" event={"ID":"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e","Type":"ContainerStarted","Data":"cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0"} Oct 31 00:31:41.939551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:41.939517 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" event={"ID":"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e","Type":"ContainerStarted","Data":"baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92"} Oct 31 00:31:41.939946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:41.939720 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:41.941063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:41.941035 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.149:8083/ready\": dial tcp 10.132.0.149:8083: connect: connection refused" Oct 31 00:31:41.970927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:41.970886 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" podStartSLOduration=1.918715908 podStartE2EDuration="11.970875361s" totalImagesPullingTime="10.052159453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:41.966407566 +0000 UTC m=+1986.027182260" watchObservedRunningTime="2026-10-31 00:31:41.970875361 +0000 UTC m=+1986.031650045" Oct 31 00:31:42.945626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:42.945589 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:42.947341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:42.947317 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:42.999092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:31:42.999070 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:31:43.089557 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:31:43.089521 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:32:13.085491 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:32:13.085457 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:32:36.716777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:36.716741 5073 scope.go:122] "RemoveContainer" containerID="c627fada6b06bafcff74aab058980e55824a0880eb32055e17a28b8190d09489" Oct 31 00:32:36.725806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:36.725773 5073 scope.go:122] "RemoveContainer" containerID="003e94064fa90f9abae1ea3e4a7a8aa2b9217fb22ba5ca8a0e742303645c87f9" Oct 31 00:32:36.733834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:36.733814 5073 scope.go:122] "RemoveContainer" containerID="8ac4f5d2ecb23f9dae1108dcb64925639e908269ff6e6b8e60b9f71a68cad2b3" Oct 31 00:32:36.741658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:36.741641 5073 scope.go:122] "RemoveContainer" containerID="8caa146a1c6524547c3e0cb5ae2d34d0763225d08dce8c3446efd2da9b7768d6" Oct 31 00:32:36.749333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:36.749315 5073 scope.go:122] "RemoveContainer" containerID="c499cb020bf82d66bb8a869ff927edb89f7d0e09b9a0b00a0c923854c667b0a5" Oct 31 00:32:36.756868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:36.756851 5073 scope.go:122] "RemoveContainer" containerID="917a92164e28d09bcd81c6cc33169554933c7a298dc978a2d262ff9a1f7e3189" Oct 31 00:32:43.085337 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:32:43.085290 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:32:51.585409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:51.585376 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:32:51.585850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:51.585551 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="49d439e5-d488-4132-a59f-85302ba62a44" containerName="otc-container" containerID="cri-o://a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2" gracePeriod=30 Oct 31 00:32:51.833513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:51.833476 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp"] Oct 31 00:32:51.833829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:51.833762 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway" containerID="cri-o://cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0" gracePeriod=30 Oct 31 00:32:51.834579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:51.834185 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway-opa" containerID="cri-o://baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92" gracePeriod=30 Oct 31 00:32:51.963180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:51.963152 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:32:52.058404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.058374 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal\" (UniqueName: \"kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal\") pod \"49d439e5-d488-4132-a59f-85302ba62a44\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " Oct 31 00:32:52.058561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.058415 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs\") pod \"49d439e5-d488-4132-a59f-85302ba62a44\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " Oct 31 00:32:52.058619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.058600 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf\" (UniqueName: \"kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf\") pod \"49d439e5-d488-4132-a59f-85302ba62a44\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " Oct 31 00:32:52.058701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.058685 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs\") pod \"49d439e5-d488-4132-a59f-85302ba62a44\" (UID: \"49d439e5-d488-4132-a59f-85302ba62a44\") " Oct 31 00:32:52.058765 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.058746 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal" pod "49d439e5-d488-4132-a59f-85302ba62a44" (UID: "49d439e5-d488-4132-a59f-85302ba62a44"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:52.058940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.058923 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/49d439e5-d488-4132-a59f-85302ba62a44-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.060685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.060660 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs" pod "49d439e5-d488-4132-a59f-85302ba62a44" (UID: "49d439e5-d488-4132-a59f-85302ba62a44"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:52.060963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.060942 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs" pod "49d439e5-d488-4132-a59f-85302ba62a44" (UID: "49d439e5-d488-4132-a59f-85302ba62a44"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:52.061021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.060963 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf" pod "49d439e5-d488-4132-a59f-85302ba62a44" (UID: "49d439e5-d488-4132-a59f-85302ba62a44"). InnerVolumeSpecName "kube-api-access-b9fhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:52.159558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.159498 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.159558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.159521 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fhf\" (UniqueName: \"kubernetes.io/projected/49d439e5-d488-4132-a59f-85302ba62a44-kube-api-access-b9fhf\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.159558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.159530 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/49d439e5-d488-4132-a59f-85302ba62a44-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.310697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.310665 5073 generic.go:356] "Generic (PLEG): container finished" podID="49d439e5-d488-4132-a59f-85302ba62a44" containerID="a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2" exitCode=0 Oct 31 00:32:52.310861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.310730 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:32:52.310861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.310754 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49d439e5-d488-4132-a59f-85302ba62a44","Type":"ContainerDied","Data":"a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2"} Oct 31 00:32:52.311001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.310864 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"49d439e5-d488-4132-a59f-85302ba62a44","Type":"ContainerDied","Data":"eda55b92613df9760520591c4c2e84b9d7810228360ff29dcda75a3b9b85b05a"} Oct 31 00:32:52.311001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.310898 5073 scope.go:122] "RemoveContainer" containerID="a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2" Oct 31 00:32:52.312937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.312912 5073 generic.go:356] "Generic (PLEG): container finished" podID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerID="baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92" exitCode=0 Oct 31 00:32:52.313029 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.312959 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" event={"ID":"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e","Type":"ContainerDied","Data":"baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92"} Oct 31 00:32:52.320977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.320945 5073 scope.go:122] "RemoveContainer" containerID="a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2" Oct 31 00:32:52.321212 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:32:52.321192 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2\": container with ID starting with a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2 not found: ID does not exist" containerID="a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2" Oct 31 00:32:52.321272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.321219 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2"} err="failed to get container status \"a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2\": rpc error: code = NotFound desc = could not find container \"a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2\": container with ID starting with a4cf946fe63f1dce3d6505925dec021bb39977e263111ebc68311b6b1d71b7d2 not found: ID does not exist" Oct 31 00:32:52.334429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.334408 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:32:52.340960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.340940 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:32:52.946132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.946093 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.149:8081/ready\": dial tcp 10.132.0.149:8081: connect: connection refused" Oct 31 00:32:52.946471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:52.946155 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.149:8083/ready\": dial tcp 10.132.0.149:8083: connect: connection refused" Oct 31 00:32:53.090745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.090721 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:32:53.166256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166191 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7\" (UniqueName: \"kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166224 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166262 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166280 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166299 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166342 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166428 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant\") pod \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\" (UID: \"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e\") " Oct 31 00:32:53.166931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166892 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.167042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.166941 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.167190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.167166 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.168366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.168344 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "kube-api-access-xggs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:53.168676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.168654 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:53.168824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.168802 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:53.173854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.173831 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant" pod "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" (UID: "d16840ea-0cbd-424f-8d07-f7f57eaa3d3e"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:53.267212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267187 5073 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tenant\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.267212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267208 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xggs7\" (UniqueName: \"kubernetes.io/projected/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-kube-api-access-xggs7\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.267341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267218 5073 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-cabundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.267341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267228 5073 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.267341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267239 5073 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-serving-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.267341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267247 5073 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-rbac\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.267341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.267256 5073 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.320389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.320355 5073 generic.go:356] "Generic (PLEG): container finished" podID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerID="cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0" exitCode=0 Oct 31 00:32:53.320485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.320386 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" event={"ID":"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e","Type":"ContainerDied","Data":"cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0"} Oct 31 00:32:53.320485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.320406 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" Oct 31 00:32:53.320485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.320447 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp" event={"ID":"d16840ea-0cbd-424f-8d07-f7f57eaa3d3e","Type":"ContainerDied","Data":"fb42911e201429cf3a3a98b7def23a8078ee7b2c93689626cccf0761ae0747c0"} Oct 31 00:32:53.320485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.320478 5073 scope.go:122] "RemoveContainer" containerID="baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92" Oct 31 00:32:53.329181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.329165 5073 scope.go:122] "RemoveContainer" containerID="cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0" Oct 31 00:32:53.337258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.337242 5073 scope.go:122] "RemoveContainer" containerID="baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92" Oct 31 00:32:53.337516 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:32:53.337498 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92\": container with ID starting with baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92 not found: ID does not exist" containerID="baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92" Oct 31 00:32:53.337561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.337523 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92"} err="failed to get container status \"baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92\": rpc error: code = NotFound desc = could not find container \"baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92\": container with ID starting with baaf3349f1effa718607e8b96a631ff9a1f74de9a80d3b71f2947968c1356d92 not found: ID does not exist" Oct 31 00:32:53.337561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.337541 5073 scope.go:122] "RemoveContainer" containerID="cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0" Oct 31 00:32:53.337775 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:32:53.337756 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0\": container with ID starting with cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0 not found: ID does not exist" containerID="cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0" Oct 31 00:32:53.337834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.337807 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0"} err="failed to get container status \"cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0\": rpc error: code = NotFound desc = could not find container \"cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0\": container with ID starting with cf0992f10d29b620d399afc65c2ec88cfac971ca15a26a6fdd4f40efbaeb20f0 not found: ID does not exist" Oct 31 00:32:53.343633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.343613 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp"] Oct 31 00:32:53.348795 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.348760 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6fd7b5dfb9-77fkp"] Oct 31 00:32:53.456928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.456883 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:32:53.461289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:53.461270 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:32:54.057720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:54.057687 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d439e5-d488-4132-a59f-85302ba62a44" path="/var/lib/kubelet/pods/49d439e5-d488-4132-a59f-85302ba62a44/volumes" Oct 31 00:32:54.058088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:54.058071 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" path="/var/lib/kubelet/pods/d16840ea-0cbd-424f-8d07-f7f57eaa3d3e/volumes" Oct 31 00:32:54.058392 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:32:54.058379 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea03fe6-c2c6-4a2e-a9fd-10e101e68d99" path="/var/lib/kubelet/pods/eea03fe6-c2c6-4a2e-a9fd-10e101e68d99/volumes" Oct 31 00:33:13.085006 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:33:13.084966 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:33:30.285033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.284998 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285317 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway-opa" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285328 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway-opa" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285349 5073 cpu_manager.go:414] "containerMap: removing container" podUID="49d439e5-d488-4132-a59f-85302ba62a44" containerName="otc-container" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285354 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d439e5-d488-4132-a59f-85302ba62a44" containerName="otc-container" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285360 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285365 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285434 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway-opa" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285444 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d439e5-d488-4132-a59f-85302ba62a44" containerName="otc-container" Oct 31 00:33:30.285462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.285450 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16840ea-0cbd-424f-8d07-f7f57eaa3d3e" containerName="tempo-gateway" Oct 31 00:33:30.288056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.288037 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:30.295214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.295186 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:33:30.295352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.295225 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:33:30.295352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.295282 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:33:30.296777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.296753 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 31 00:33:30.337869 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.337848 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p8r\" (UniqueName: \"kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r\") pod \"fake-gcs-bucket-creator\" (UID: \"00305637-3313-4372-ad93-9deeec83353c\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:30.438827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.438804 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p8r\" (UniqueName: \"kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r\") pod \"fake-gcs-bucket-creator\" (UID: \"00305637-3313-4372-ad93-9deeec83353c\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:30.452315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.452296 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p8r\" (UniqueName: \"kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r\") pod \"fake-gcs-bucket-creator\" (UID: \"00305637-3313-4372-ad93-9deeec83353c\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:30.601272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.601219 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:30.872450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:30.872367 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 31 00:33:30.876277 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:33:30.876249 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00305637_3313_4372_ad93_9deeec83353c.slice/crio-e37a494657e85cd705a34353f13339080a0b23929decec7dfea878201075ba73 WatchSource:0}: Error finding container e37a494657e85cd705a34353f13339080a0b23929decec7dfea878201075ba73: Status 404 returned error can't find the container with id e37a494657e85cd705a34353f13339080a0b23929decec7dfea878201075ba73 Oct 31 00:33:31.515695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:31.515662 5073 generic.go:356] "Generic (PLEG): container finished" podID="00305637-3313-4372-ad93-9deeec83353c" containerID="f09d65b04557b2865188a43fea3bda05254efaf5c698fbffc68c9ecacd63866d" exitCode=0 Oct 31 00:33:31.516055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:31.515749 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"00305637-3313-4372-ad93-9deeec83353c","Type":"ContainerDied","Data":"f09d65b04557b2865188a43fea3bda05254efaf5c698fbffc68c9ecacd63866d"} Oct 31 00:33:31.516055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:31.515807 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"00305637-3313-4372-ad93-9deeec83353c","Type":"ContainerStarted","Data":"e37a494657e85cd705a34353f13339080a0b23929decec7dfea878201075ba73"} Oct 31 00:33:32.647640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:32.647617 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:32.759113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:32.759075 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r\" (UniqueName: \"kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r\") pod \"00305637-3313-4372-ad93-9deeec83353c\" (UID: \"00305637-3313-4372-ad93-9deeec83353c\") " Oct 31 00:33:32.761361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:32.761325 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r" pod "00305637-3313-4372-ad93-9deeec83353c" (UID: "00305637-3313-4372-ad93-9deeec83353c"). InnerVolumeSpecName "kube-api-access-98p8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:32.860160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:32.860104 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98p8r\" (UniqueName: \"kubernetes.io/projected/00305637-3313-4372-ad93-9deeec83353c-kube-api-access-98p8r\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:33.527404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:33.527372 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" Oct 31 00:33:33.527602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:33.527365 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"00305637-3313-4372-ad93-9deeec83353c","Type":"ContainerDied","Data":"e37a494657e85cd705a34353f13339080a0b23929decec7dfea878201075ba73"} Oct 31 00:33:33.527602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:33.527503 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37a494657e85cd705a34353f13339080a0b23929decec7dfea878201075ba73" Oct 31 00:33:35.442696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.442667 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:33:35.443105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.443011 5073 cpu_manager.go:414] "containerMap: removing container" podUID="00305637-3313-4372-ad93-9deeec83353c" containerName="create-bucket" Oct 31 00:33:35.443105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.443024 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00305637-3313-4372-ad93-9deeec83353c" containerName="create-bucket" Oct 31 00:33:35.443178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.443117 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="00305637-3313-4372-ad93-9deeec83353c" containerName="create-bucket" Oct 31 00:33:35.446273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.446250 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.454707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.454684 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:33:35.455840 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.455815 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:33:35.455972 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.455917 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w5bd4\"" Oct 31 00:33:35.458924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.458904 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:33:35.466880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.466860 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:33:35.466880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.466877 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:33:35.467028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.466939 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:33:35.580566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.580537 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.580704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.580574 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.580704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.580599 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snl8\" (UniqueName: \"kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.580886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.580741 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.681836 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.681774 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.681992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.681851 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.681992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.681871 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snl8\" (UniqueName: \"kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.681992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.681906 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.682557 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.682532 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.684421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.684393 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.684545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.684424 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.699571 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.699517 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snl8\" (UniqueName: \"kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8\") pod \"data-science-collector-collector-0\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:35.762525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:35.762503 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:33:36.038916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.038747 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:33:36.041533 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:33:36.041512 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc314ad5a_eefe_446a_8d46_1cb835f7b9da.slice/crio-896e416f5e8d685fcc04f3ce80bafb59fb79ea82d16c7751dfb53d8a4f0f51ef WatchSource:0}: Error finding container 896e416f5e8d685fcc04f3ce80bafb59fb79ea82d16c7751dfb53d8a4f0f51ef: Status 404 returned error can't find the container with id 896e416f5e8d685fcc04f3ce80bafb59fb79ea82d16c7751dfb53d8a4f0f51ef Oct 31 00:33:36.043429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.043400 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:33:36.043494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.043455 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:33:36.282765 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.282673 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:33:36.299754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.288763 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:33:36.299754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.290258 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:33:36.299754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.296450 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:33:36.549337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.549307 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c314ad5a-eefe-446a-8d46-1cb835f7b9da","Type":"ContainerStarted","Data":"9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10"} Oct 31 00:33:36.549703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.549364 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c314ad5a-eefe-446a-8d46-1cb835f7b9da","Type":"ContainerStarted","Data":"896e416f5e8d685fcc04f3ce80bafb59fb79ea82d16c7751dfb53d8a4f0f51ef"} Oct 31 00:33:36.570119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.570010 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.569993078 podStartE2EDuration="1.569993078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:36.569639934 +0000 UTC m=+2100.630414631" watchObservedRunningTime="2026-10-31 00:33:36.569993078 +0000 UTC m=+2100.630767774" Oct 31 00:33:36.796001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.795974 5073 scope.go:122] "RemoveContainer" containerID="2dd94258ad6dbc8f23dd4a0168b2a2f1216340d5aeec48ae87ac870bda400a9a" Oct 31 00:33:36.804447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.804428 5073 scope.go:122] "RemoveContainer" containerID="781b7418f3f0ef2390828544e4a637e1bf8f3a6d4802bd27b6638f5b0aef2bb9" Oct 31 00:33:36.811778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.811763 5073 scope.go:122] "RemoveContainer" containerID="22d871307738030f4913849499f0496b4b7cd5801e95f2d9c3ea3b046efe9a94" Oct 31 00:33:36.819047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.819029 5073 scope.go:122] "RemoveContainer" containerID="7b75fccca295683c9010015457e26146b1a8a34d61547889930d0a5d9ea1510c" Oct 31 00:33:36.826499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.826482 5073 scope.go:122] "RemoveContainer" containerID="b45f5fd897bf052622bba2c2d91deab81d36987154fe910616dfa18db6b890c5" Oct 31 00:33:36.833967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.833952 5073 scope.go:122] "RemoveContainer" containerID="695c265b74ffb15a37706d283246ace196342b5aa518076d9c807ae5579b4899" Oct 31 00:33:36.843205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.843178 5073 scope.go:122] "RemoveContainer" containerID="72d3c91d46156a9e908ef780a2e4e044e7d32775b7feda69ad7969fcd6160245" Oct 31 00:33:36.850339 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.850323 5073 scope.go:122] "RemoveContainer" containerID="f5610b97b2dd508b6039e01d6ccec08308f968e9b90aa5bec1e29e176fd2d095" Oct 31 00:33:36.857313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.857296 5073 scope.go:122] "RemoveContainer" containerID="e24f32aeb81ad38b299dd1b7bc8ab6e5e3adc5d5facfe4a6088bf3bd37948d7b" Oct 31 00:33:36.864413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.864389 5073 scope.go:122] "RemoveContainer" containerID="9fbdf3b21489ac283e0c825b75b27f5b53b1e979530fd1037aeb69b78aada0a6" Oct 31 00:33:36.871206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.871187 5073 scope.go:122] "RemoveContainer" containerID="1e279ff2317677df1cc816e1d52656e40c376d316d3997411098c4f05b141d1f" Oct 31 00:33:36.877985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:36.877968 5073 scope.go:122] "RemoveContainer" containerID="b271d23a4996a1fa09cd0a4ab710971d91667e835d9507a8d8587cf5c525d095" Oct 31 00:33:38.416349 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.416313 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn"] Oct 31 00:33:38.422653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.422628 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.427016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.426991 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:33:38.428071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.427972 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-rnszp\"" Oct 31 00:33:38.428071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.427994 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:33:38.428071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.427978 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 31 00:33:38.428071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.428023 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 31 00:33:38.428071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.428036 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 31 00:33:38.428071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.428071 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 31 00:33:38.436915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.435526 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn"] Oct 31 00:33:38.508603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508571 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.508767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508610 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflvz\" (UniqueName: \"kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.508767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508644 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.508881 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508761 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.508881 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508811 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.508946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508899 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.508982 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.508947 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.609675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609644 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.609675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609680 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.609951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609705 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.609951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609846 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.609951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609902 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.609951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609940 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflvz\" (UniqueName: \"kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.610167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.609999 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.610497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.610469 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.610816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.610697 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.610816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.610765 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.612535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.612508 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.612609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.612580 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.613037 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.613013 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.627170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.627149 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflvz\" (UniqueName: \"kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz\") pod \"tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:38.741292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:38.741263 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:39.025337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.025260 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn"] Oct 31 00:33:39.028393 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:33:39.028367 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c4f572_a702_4e94_b952_25ac48facdd7.slice/crio-c65b893575ca16be64421abea1fd1f4781fe509584d9498815791de53f2b0f04 WatchSource:0}: Error finding container c65b893575ca16be64421abea1fd1f4781fe509584d9498815791de53f2b0f04: Status 404 returned error can't find the container with id c65b893575ca16be64421abea1fd1f4781fe509584d9498815791de53f2b0f04 Oct 31 00:33:39.565357 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.565292 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" event={"ID":"02c4f572-a702-4e94-b952-25ac48facdd7","Type":"ContainerStarted","Data":"3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2"} Oct 31 00:33:39.565357 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.565352 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:39.565845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.565377 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" event={"ID":"02c4f572-a702-4e94-b952-25ac48facdd7","Type":"ContainerStarted","Data":"3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8"} Oct 31 00:33:39.565845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.565409 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:39.565845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.565433 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" event={"ID":"02c4f572-a702-4e94-b952-25ac48facdd7","Type":"ContainerStarted","Data":"c65b893575ca16be64421abea1fd1f4781fe509584d9498815791de53f2b0f04"} Oct 31 00:33:39.566878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.566852 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.152:8081/ready\": dial tcp 10.132.0.152:8081: connect: connection refused" Oct 31 00:33:39.566945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.566860 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.152:8083/ready\": dial tcp 10.132.0.152:8083: connect: connection refused" Oct 31 00:33:39.590747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:39.590702 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" podStartSLOduration=1.590690583 podStartE2EDuration="1.590690583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:39.588927817 +0000 UTC m=+2103.649702511" watchObservedRunningTime="2026-10-31 00:33:39.590690583 +0000 UTC m=+2103.651465277" Oct 31 00:33:40.570563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:40.570522 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.152:8083/ready\": dial tcp 10.132.0.152:8083: connect: connection refused" Oct 31 00:33:41.100566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:41.100539 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:41.576988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:41.576960 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:42.123093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:42.123061 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn"] Oct 31 00:33:43.085516 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:33:43.085481 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:33:43.584323 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:43.584287 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway" containerID="cri-o://3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8" gracePeriod=30 Oct 31 00:33:43.584496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:43.584286 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway-opa" containerID="cri-o://3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2" gracePeriod=30 Oct 31 00:33:44.590703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.590670 5073 generic.go:356] "Generic (PLEG): container finished" podID="02c4f572-a702-4e94-b952-25ac48facdd7" containerID="3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2" exitCode=0 Oct 31 00:33:44.591176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.590744 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" event={"ID":"02c4f572-a702-4e94-b952-25ac48facdd7","Type":"ContainerDied","Data":"3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2"} Oct 31 00:33:44.838489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.838462 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:44.965489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965419 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965459 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965487 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz\" (UniqueName: \"kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965581 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965678 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965744 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965820 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls\") pod \"02c4f572-a702-4e94-b952-25ac48facdd7\" (UID: \"02c4f572-a702-4e94-b952-25ac48facdd7\") " Oct 31 00:33:44.965992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965873 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:44.966052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.965997 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:44.966115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.966046 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:44.966185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.966164 5073 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-rbac\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:44.966241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.966195 5073 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:44.966286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.966234 5073 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/02c4f572-a702-4e94-b952-25ac48facdd7-cabundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:44.967899 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.967879 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:44.968311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.968289 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "kube-api-access-bflvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:44.968384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.968363 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:44.972097 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:44.972079 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant" pod "02c4f572-a702-4e94-b952-25ac48facdd7" (UID: "02c4f572-a702-4e94-b952-25ac48facdd7"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:45.067550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.067525 5073 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tenant\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:45.067550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.067549 5073 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:45.067687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.067565 5073 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/02c4f572-a702-4e94-b952-25ac48facdd7-serving-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:45.067687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.067579 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflvz\" (UniqueName: \"kubernetes.io/projected/02c4f572-a702-4e94-b952-25ac48facdd7-kube-api-access-bflvz\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:45.597695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.597618 5073 generic.go:356] "Generic (PLEG): container finished" podID="02c4f572-a702-4e94-b952-25ac48facdd7" containerID="3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8" exitCode=0 Oct 31 00:33:45.597695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.597673 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" Oct 31 00:33:45.598169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.597692 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" event={"ID":"02c4f572-a702-4e94-b952-25ac48facdd7","Type":"ContainerDied","Data":"3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8"} Oct 31 00:33:45.598169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.597751 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn" event={"ID":"02c4f572-a702-4e94-b952-25ac48facdd7","Type":"ContainerDied","Data":"c65b893575ca16be64421abea1fd1f4781fe509584d9498815791de53f2b0f04"} Oct 31 00:33:45.598169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.597776 5073 scope.go:122] "RemoveContainer" containerID="3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2" Oct 31 00:33:45.607178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.607161 5073 scope.go:122] "RemoveContainer" containerID="3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8" Oct 31 00:33:45.614819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.614802 5073 scope.go:122] "RemoveContainer" containerID="3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2" Oct 31 00:33:45.615062 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:33:45.615045 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2\": container with ID starting with 3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2 not found: ID does not exist" containerID="3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2" Oct 31 00:33:45.615111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.615072 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2"} err="failed to get container status \"3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2\": rpc error: code = NotFound desc = could not find container \"3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2\": container with ID starting with 3b45edecc7180b658d0db3f7d9f5e06033dea8813d2f32e1988090e6fb42f1c2 not found: ID does not exist" Oct 31 00:33:45.615111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.615092 5073 scope.go:122] "RemoveContainer" containerID="3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8" Oct 31 00:33:45.615307 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:33:45.615291 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8\": container with ID starting with 3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8 not found: ID does not exist" containerID="3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8" Oct 31 00:33:45.615348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.615311 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8"} err="failed to get container status \"3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8\": rpc error: code = NotFound desc = could not find container \"3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8\": container with ID starting with 3d61e54cc3757a940ef872fcbf710c68c2d6a1c30b6fbe1bb1fe861f3332d1d8 not found: ID does not exist" Oct 31 00:33:45.621835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.621814 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn"] Oct 31 00:33:45.628734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:45.628713 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6c5dbbb858-5lmpn"] Oct 31 00:33:46.058846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:33:46.058819 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" path="/var/lib/kubelet/pods/02c4f572-a702-4e94-b952-25ac48facdd7/volumes" Oct 31 00:34:13.085567 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:34:13.085534 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:34:36.953428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.953397 5073 scope.go:122] "RemoveContainer" containerID="d26507177cb73fd31ebd150e346733a392a4be7592b84516f25a74b42e0be2a9" Oct 31 00:34:36.962021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.962000 5073 scope.go:122] "RemoveContainer" containerID="5c9833fa2742ce62723a18d3b477f8a5ba3647cc9c7177746ab67b5194fedc3d" Oct 31 00:34:36.969870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.969850 5073 scope.go:122] "RemoveContainer" containerID="cd574df4f1fb37cb366e5c818ee148523d0979b15aadcfb3e009544ebe07a9d5" Oct 31 00:34:36.977279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.977263 5073 scope.go:122] "RemoveContainer" containerID="88c4da1c36677edf90b6d18310851dda19960ec7c433ce170efc8510bd087831" Oct 31 00:34:36.984269 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.984250 5073 scope.go:122] "RemoveContainer" containerID="76fed8aaf65c10967069d2245b5f4e6fbd6fd4b6e732d9d9b6ab95fc08ae9017" Oct 31 00:34:36.991211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.991194 5073 scope.go:122] "RemoveContainer" containerID="38c31d54de1ff7d935a6d29eb1cc93588f061054f44288b742cb5964b4c52bca" Oct 31 00:34:36.998148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:36.998131 5073 scope.go:122] "RemoveContainer" containerID="80912767e5ddfa6376f475714eee5e255875147ac4f18d56b0b6570157567991" Oct 31 00:34:43.085137 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:34:43.085102 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:34:54.314864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.314835 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:34:54.315379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.315047 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c314ad5a-eefe-446a-8d46-1cb835f7b9da" containerName="otc-container" containerID="cri-o://9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10" gracePeriod=30 Oct 31 00:34:54.564738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.564709 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:54.664628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.664565 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs\") pod \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " Oct 31 00:34:54.664628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.664604 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8\" (UniqueName: \"kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8\") pod \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " Oct 31 00:34:54.664833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.664641 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal\") pod \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " Oct 31 00:34:54.664833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.664682 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs\") pod \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\" (UID: \"c314ad5a-eefe-446a-8d46-1cb835f7b9da\") " Oct 31 00:34:54.665083 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.665051 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal" pod "c314ad5a-eefe-446a-8d46-1cb835f7b9da" (UID: "c314ad5a-eefe-446a-8d46-1cb835f7b9da"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:54.666933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.666910 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs" pod "c314ad5a-eefe-446a-8d46-1cb835f7b9da" (UID: "c314ad5a-eefe-446a-8d46-1cb835f7b9da"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:54.667017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.666942 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs" pod "c314ad5a-eefe-446a-8d46-1cb835f7b9da" (UID: "c314ad5a-eefe-446a-8d46-1cb835f7b9da"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:54.667055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.667036 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8" pod "c314ad5a-eefe-446a-8d46-1cb835f7b9da" (UID: "c314ad5a-eefe-446a-8d46-1cb835f7b9da"). InnerVolumeSpecName "kube-api-access-5snl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:54.765389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.765356 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:54.765389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.765387 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:54.765389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.765396 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c314ad5a-eefe-446a-8d46-1cb835f7b9da-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:54.765562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.765405 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5snl8\" (UniqueName: \"kubernetes.io/projected/c314ad5a-eefe-446a-8d46-1cb835f7b9da-kube-api-access-5snl8\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:54.926774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.926700 5073 generic.go:356] "Generic (PLEG): container finished" podID="c314ad5a-eefe-446a-8d46-1cb835f7b9da" containerID="9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10" exitCode=0 Oct 31 00:34:54.926774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.926742 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:54.926946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.926809 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c314ad5a-eefe-446a-8d46-1cb835f7b9da","Type":"ContainerDied","Data":"9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10"} Oct 31 00:34:54.926946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.926863 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c314ad5a-eefe-446a-8d46-1cb835f7b9da","Type":"ContainerDied","Data":"896e416f5e8d685fcc04f3ce80bafb59fb79ea82d16c7751dfb53d8a4f0f51ef"} Oct 31 00:34:54.926946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.926885 5073 scope.go:122] "RemoveContainer" containerID="9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10" Oct 31 00:34:54.935507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.935492 5073 scope.go:122] "RemoveContainer" containerID="9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10" Oct 31 00:34:54.935757 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:34:54.935737 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10\": container with ID starting with 9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10 not found: ID does not exist" containerID="9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10" Oct 31 00:34:54.935833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.935772 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10"} err="failed to get container status \"9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10\": rpc error: code = NotFound desc = could not find container \"9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10\": container with ID starting with 9971b09f9b515efb6bb144bd150ce06812727d497ecb1c670075565e2d2d7a10 not found: ID does not exist" Oct 31 00:34:54.947898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.947877 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:34:54.952315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.952295 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:34:54.977433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977410 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:34:54.977719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977707 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c314ad5a-eefe-446a-8d46-1cb835f7b9da" containerName="otc-container" Oct 31 00:34:54.977770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977720 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c314ad5a-eefe-446a-8d46-1cb835f7b9da" containerName="otc-container" Oct 31 00:34:54.977770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977731 5073 cpu_manager.go:414] "containerMap: removing container" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway" Oct 31 00:34:54.977770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977737 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway" Oct 31 00:34:54.977770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977754 5073 cpu_manager.go:414] "containerMap: removing container" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway-opa" Oct 31 00:34:54.977770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977759 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway-opa" Oct 31 00:34:54.977961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977836 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway-opa" Oct 31 00:34:54.977961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977848 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c314ad5a-eefe-446a-8d46-1cb835f7b9da" containerName="otc-container" Oct 31 00:34:54.977961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.977854 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c4f572-a702-4e94-b952-25ac48facdd7" containerName="tempo-gateway" Oct 31 00:34:54.982027 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.982011 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:54.985926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.985909 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:34:54.985926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.985916 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:34:54.986082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.985970 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w5bd4\"" Oct 31 00:34:54.986082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.986020 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:34:54.986082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.986024 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:34:54.986082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.986052 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:34:54.990922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:54.990901 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:34:55.068173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.068147 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzz6\" (UniqueName: \"kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.068292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.068184 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.068292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.068208 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.068429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.068411 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.169010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.168984 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.169129 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.169062 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.169129 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.169109 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzz6\" (UniqueName: \"kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.169247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.169139 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.169656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.169636 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.171701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.171674 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.171813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.171713 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.181506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.181461 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzz6\" (UniqueName: \"kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6\") pod \"data-science-collector-collector-0\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.295391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.295361 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:34:55.570303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.570274 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:34:55.573777 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:34:55.573751 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c088f5_97f2_4d59_987e_d70774ec85a7.slice/crio-9f496a3a9aaec63f5324d1ec30749a7f535fa1c310bf8931b25deeb73231b095 WatchSource:0}: Error finding container 9f496a3a9aaec63f5324d1ec30749a7f535fa1c310bf8931b25deeb73231b095: Status 404 returned error can't find the container with id 9f496a3a9aaec63f5324d1ec30749a7f535fa1c310bf8931b25deeb73231b095 Oct 31 00:34:55.576168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.576124 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:34:55.576249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.576206 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:34:55.934230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.934151 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c9c088f5-97f2-4d59-987e-d70774ec85a7","Type":"ContainerStarted","Data":"45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e"} Oct 31 00:34:55.934230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.934198 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c9c088f5-97f2-4d59-987e-d70774ec85a7","Type":"ContainerStarted","Data":"9f496a3a9aaec63f5324d1ec30749a7f535fa1c310bf8931b25deeb73231b095"} Oct 31 00:34:55.954302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:55.954253 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.954233616 podStartE2EDuration="1.954233616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:55.952870051 +0000 UTC m=+2180.013644746" watchObservedRunningTime="2026-10-31 00:34:55.954233616 +0000 UTC m=+2180.015008311" Oct 31 00:34:56.059242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:34:56.058829 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c314ad5a-eefe-446a-8d46-1cb835f7b9da" path="/var/lib/kubelet/pods/c314ad5a-eefe-446a-8d46-1cb835f7b9da/volumes" Oct 31 00:35:13.084702 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:35:13.084669 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:35:24.037036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.036952 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:24.037504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.037204 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c9c088f5-97f2-4d59-987e-d70774ec85a7" containerName="otc-container" containerID="cri-o://45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e" gracePeriod=30 Oct 31 00:35:24.318532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.318506 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:24.502085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.502050 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal\") pod \"c9c088f5-97f2-4d59-987e-d70774ec85a7\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " Oct 31 00:35:24.502271 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.502146 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs\") pod \"c9c088f5-97f2-4d59-987e-d70774ec85a7\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " Oct 31 00:35:24.502271 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.502199 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6\" (UniqueName: \"kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6\") pod \"c9c088f5-97f2-4d59-987e-d70774ec85a7\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " Oct 31 00:35:24.502271 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.502229 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs\") pod \"c9c088f5-97f2-4d59-987e-d70774ec85a7\" (UID: \"c9c088f5-97f2-4d59-987e-d70774ec85a7\") " Oct 31 00:35:24.502463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.502372 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal" pod "c9c088f5-97f2-4d59-987e-d70774ec85a7" (UID: "c9c088f5-97f2-4d59-987e-d70774ec85a7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:24.502566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.502542 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c9c088f5-97f2-4d59-987e-d70774ec85a7-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.504411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.504377 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6" pod "c9c088f5-97f2-4d59-987e-d70774ec85a7" (UID: "c9c088f5-97f2-4d59-987e-d70774ec85a7"). InnerVolumeSpecName "kube-api-access-tnzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:24.504580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.504410 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs" pod "c9c088f5-97f2-4d59-987e-d70774ec85a7" (UID: "c9c088f5-97f2-4d59-987e-d70774ec85a7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:24.504580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.504439 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs" pod "c9c088f5-97f2-4d59-987e-d70774ec85a7" (UID: "c9c088f5-97f2-4d59-987e-d70774ec85a7"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:24.603140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.603070 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzz6\" (UniqueName: \"kubernetes.io/projected/c9c088f5-97f2-4d59-987e-d70774ec85a7-kube-api-access-tnzz6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.603140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.603104 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.603140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:24.603114 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c088f5-97f2-4d59-987e-d70774ec85a7-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.064505 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.064472 5073 generic.go:356] "Generic (PLEG): container finished" podID="c9c088f5-97f2-4d59-987e-d70774ec85a7" containerID="45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e" exitCode=0 Oct 31 00:35:25.064987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.064518 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:25.064987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.064558 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c9c088f5-97f2-4d59-987e-d70774ec85a7","Type":"ContainerDied","Data":"45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e"} Oct 31 00:35:25.064987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.064610 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c9c088f5-97f2-4d59-987e-d70774ec85a7","Type":"ContainerDied","Data":"9f496a3a9aaec63f5324d1ec30749a7f535fa1c310bf8931b25deeb73231b095"} Oct 31 00:35:25.064987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.064625 5073 scope.go:122] "RemoveContainer" containerID="45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e" Oct 31 00:35:25.074092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.074075 5073 scope.go:122] "RemoveContainer" containerID="45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e" Oct 31 00:35:25.074382 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:35:25.074365 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e\": container with ID starting with 45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e not found: ID does not exist" containerID="45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e" Oct 31 00:35:25.074437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.074390 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e"} err="failed to get container status \"45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e\": rpc error: code = NotFound desc = could not find container \"45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e\": container with ID starting with 45b418882941dcb1bed2ea1c4687eff2ec117c7043f9a22c9448ac919c05e23e not found: ID does not exist" Oct 31 00:35:25.086937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.086900 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:25.091676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.091650 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:25.872610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.872579 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 31 00:35:25.874619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:25.874596 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"] Oct 31 00:35:26.058155 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:26.058120 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00305637-3313-4372-ad93-9deeec83353c" path="/var/lib/kubelet/pods/00305637-3313-4372-ad93-9deeec83353c/volumes" Oct 31 00:35:26.058510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:26.058497 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c088f5-97f2-4d59-987e-d70774ec85a7" path="/var/lib/kubelet/pods/c9c088f5-97f2-4d59-987e-d70774ec85a7/volumes" Oct 31 00:35:29.084743 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.084711 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:29.085153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.085075 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c9c088f5-97f2-4d59-987e-d70774ec85a7" containerName="otc-container" Oct 31 00:35:29.085153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.085089 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c088f5-97f2-4d59-987e-d70774ec85a7" containerName="otc-container" Oct 31 00:35:29.085236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.085178 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c088f5-97f2-4d59-987e-d70774ec85a7" containerName="otc-container" Oct 31 00:35:29.088318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.088295 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.102427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.102401 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:29.132706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.132678 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.132862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.132742 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqkp\" (UniqueName: \"kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.132862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.132821 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.132862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.132858 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.170798 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.170745 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:35:29.170925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.170838 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:35:29.170925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.170879 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:35:29.170925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.170879 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:35:29.170925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.170912 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r28ll\"" Oct 31 00:35:29.170925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.170910 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:35:29.233975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.233945 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqkp\" (UniqueName: \"kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.234084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.233994 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.234084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.234019 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.234084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.234066 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.234752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.234723 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.236469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.236443 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.236569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.236553 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.246051 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.246032 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqkp\" (UniqueName: \"kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp\") pod \"data-science-collector-collector-1\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.405127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.405079 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:29.680180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.680106 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:29.682995 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:35:29.682965 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01b2582_e8bd_4847_8a21_b69c68f5643a.slice/crio-2c376e18a4fa55ee9727c1eb8ea13e772f9079bb6f01c073a6ba4475caca13fc WatchSource:0}: Error finding container 2c376e18a4fa55ee9727c1eb8ea13e772f9079bb6f01c073a6ba4475caca13fc: Status 404 returned error can't find the container with id 2c376e18a4fa55ee9727c1eb8ea13e772f9079bb6f01c073a6ba4475caca13fc Oct 31 00:35:29.685000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.684964 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:35:29.685092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:29.685037 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:35:30.089619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:30.089586 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b01b2582-e8bd-4847-8a21-b69c68f5643a","Type":"ContainerStarted","Data":"709a0bba1c9854bd109ce29919a4da4547ac5ae6d525beb2b02f81432b1294a6"} Oct 31 00:35:30.090034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:30.089646 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b01b2582-e8bd-4847-8a21-b69c68f5643a","Type":"ContainerStarted","Data":"2c376e18a4fa55ee9727c1eb8ea13e772f9079bb6f01c073a6ba4475caca13fc"} Oct 31 00:35:30.110626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:30.110581 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.110568563 podStartE2EDuration="1.110568563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:30.109285502 +0000 UTC m=+2214.170060198" watchObservedRunningTime="2026-10-31 00:35:30.110568563 +0000 UTC m=+2214.171343261" Oct 31 00:35:43.085945 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:35:43.085906 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:35:44.908548 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:44.908519 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:44.909070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:44.908718 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b01b2582-e8bd-4847-8a21-b69c68f5643a" containerName="otc-container" containerID="cri-o://709a0bba1c9854bd109ce29919a4da4547ac5ae6d525beb2b02f81432b1294a6" gracePeriod=30 Oct 31 00:35:45.154072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.154031 5073 generic.go:356] "Generic (PLEG): container finished" podID="b01b2582-e8bd-4847-8a21-b69c68f5643a" containerID="709a0bba1c9854bd109ce29919a4da4547ac5ae6d525beb2b02f81432b1294a6" exitCode=0 Oct 31 00:35:45.154072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.154068 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b01b2582-e8bd-4847-8a21-b69c68f5643a","Type":"ContainerDied","Data":"709a0bba1c9854bd109ce29919a4da4547ac5ae6d525beb2b02f81432b1294a6"} Oct 31 00:35:45.154284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.154116 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b01b2582-e8bd-4847-8a21-b69c68f5643a","Type":"ContainerDied","Data":"2c376e18a4fa55ee9727c1eb8ea13e772f9079bb6f01c073a6ba4475caca13fc"} Oct 31 00:35:45.154284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.154137 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c376e18a4fa55ee9727c1eb8ea13e772f9079bb6f01c073a6ba4475caca13fc" Oct 31 00:35:45.166203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.166147 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:45.254245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.254216 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal\") pod \"b01b2582-e8bd-4847-8a21-b69c68f5643a\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " Oct 31 00:35:45.254245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.254254 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs\") pod \"b01b2582-e8bd-4847-8a21-b69c68f5643a\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " Oct 31 00:35:45.254437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.254306 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs\") pod \"b01b2582-e8bd-4847-8a21-b69c68f5643a\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " Oct 31 00:35:45.254437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.254353 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp\" (UniqueName: \"kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp\") pod \"b01b2582-e8bd-4847-8a21-b69c68f5643a\" (UID: \"b01b2582-e8bd-4847-8a21-b69c68f5643a\") " Oct 31 00:35:45.254588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.254564 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal" pod "b01b2582-e8bd-4847-8a21-b69c68f5643a" (UID: "b01b2582-e8bd-4847-8a21-b69c68f5643a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:45.256489 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.256459 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs" pod "b01b2582-e8bd-4847-8a21-b69c68f5643a" (UID: "b01b2582-e8bd-4847-8a21-b69c68f5643a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:45.256609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.256557 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp" pod "b01b2582-e8bd-4847-8a21-b69c68f5643a" (UID: "b01b2582-e8bd-4847-8a21-b69c68f5643a"). InnerVolumeSpecName "kube-api-access-wnqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:45.256609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.256566 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs" pod "b01b2582-e8bd-4847-8a21-b69c68f5643a" (UID: "b01b2582-e8bd-4847-8a21-b69c68f5643a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:45.355358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.355335 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnqkp\" (UniqueName: \"kubernetes.io/projected/b01b2582-e8bd-4847-8a21-b69c68f5643a-kube-api-access-wnqkp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.355358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.355359 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b01b2582-e8bd-4847-8a21-b69c68f5643a-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.355510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.355370 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.355510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:45.355378 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b01b2582-e8bd-4847-8a21-b69c68f5643a-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.159074 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:46.159046 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:46.179452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:46.179425 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:46.183386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:46.183364 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:48.057478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:48.057435 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01b2582-e8bd-4847-8a21-b69c68f5643a" path="/var/lib/kubelet/pods/b01b2582-e8bd-4847-8a21-b69c68f5643a/volumes" Oct 31 00:35:53.648373 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.648333 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk"] Oct 31 00:35:53.648862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.648685 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b01b2582-e8bd-4847-8a21-b69c68f5643a" containerName="otc-container" Oct 31 00:35:53.648862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.648698 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01b2582-e8bd-4847-8a21-b69c68f5643a" containerName="otc-container" Oct 31 00:35:53.648862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.648775 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01b2582-e8bd-4847-8a21-b69c68f5643a" containerName="otc-container" Oct 31 00:35:53.652043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.652023 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:35:53.660619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.660589 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:35:53.661523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.661500 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:35:53.661678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.661615 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:35:53.662119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.662099 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk"] Oct 31 00:35:53.717770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.717738 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8lz\" (UniqueName: \"kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk\" (UID: \"ddd9ff87-74f5-4006-8cf3-b451294222a8\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:35:53.818679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.818653 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8lz\" (UniqueName: \"kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk\" (UID: \"ddd9ff87-74f5-4006-8cf3-b451294222a8\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:35:53.831190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.831161 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8lz\" (UniqueName: \"kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk\" (UID: \"ddd9ff87-74f5-4006-8cf3-b451294222a8\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:35:53.964902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:53.964829 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:35:54.038700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.038650 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98"] Oct 31 00:35:54.046860 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.045058 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.063181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.059271 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98"] Oct 31 00:35:54.122025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.121997 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.122188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.122052 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzh6v\" (UniqueName: \"kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.122188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.122156 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.144751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.144709 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:35:54.144893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.144779 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:35:54.144893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.144847 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-gpb29\"" Oct 31 00:35:54.223218 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.223146 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.223218 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.223188 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzh6v\" (UniqueName: \"kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.223379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.223220 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.223933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.223871 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.225730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.225711 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.256441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.256416 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzh6v\" (UniqueName: \"kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v\") pod \"data-science-collector-targetallocator-7dd4b5ccd7-6dd98\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.265516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.265491 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk"] Oct 31 00:35:54.269049 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:35:54.269023 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddd9ff87_74f5_4006_8cf3_b451294222a8.slice/crio-33e92ec148447955edcd5d0e8dc8ca97e18f143b9ca0ed90e3f9a57ffd7c3810 WatchSource:0}: Error finding container 33e92ec148447955edcd5d0e8dc8ca97e18f143b9ca0ed90e3f9a57ffd7c3810: Status 404 returned error can't find the container with id 33e92ec148447955edcd5d0e8dc8ca97e18f143b9ca0ed90e3f9a57ffd7c3810 Oct 31 00:35:54.381385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.381355 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:54.655067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.654951 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98"] Oct 31 00:35:54.657604 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:35:54.657574 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaeafbb4_c94c_4efc_a44d_f7ae28847d90.slice/crio-4f115751a0d43fe19151cb080a5853566f56e32c9a288bd74a6d2031c5c39bfe WatchSource:0}: Error finding container 4f115751a0d43fe19151cb080a5853566f56e32c9a288bd74a6d2031c5c39bfe: Status 404 returned error can't find the container with id 4f115751a0d43fe19151cb080a5853566f56e32c9a288bd74a6d2031c5c39bfe Oct 31 00:35:54.659772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.659739 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:35:54.659860 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:54.659826 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:35:55.198481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.198437 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" event={"ID":"ddd9ff87-74f5-4006-8cf3-b451294222a8","Type":"ContainerStarted","Data":"c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3"} Oct 31 00:35:55.198481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.198477 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" event={"ID":"ddd9ff87-74f5-4006-8cf3-b451294222a8","Type":"ContainerStarted","Data":"33e92ec148447955edcd5d0e8dc8ca97e18f143b9ca0ed90e3f9a57ffd7c3810"} Oct 31 00:35:55.200128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.200101 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" event={"ID":"caeafbb4-c94c-4efc-a44d-f7ae28847d90","Type":"ContainerStarted","Data":"53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201"} Oct 31 00:35:55.200256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.200134 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" event={"ID":"caeafbb4-c94c-4efc-a44d-f7ae28847d90","Type":"ContainerStarted","Data":"4f115751a0d43fe19151cb080a5853566f56e32c9a288bd74a6d2031c5c39bfe"} Oct 31 00:35:55.200256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.200155 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:55.202010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.201988 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:35:55.216084 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.216046 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" podStartSLOduration=2.216033993 podStartE2EDuration="2.216033993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:55.215579448 +0000 UTC m=+2239.276354145" watchObservedRunningTime="2026-10-31 00:35:55.216033993 +0000 UTC m=+2239.276808687" Oct 31 00:35:55.233512 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:55.233464 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" podStartSLOduration=1.233445807 podStartE2EDuration="1.233445807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:55.233134228 +0000 UTC m=+2239.293908925" watchObservedRunningTime="2026-10-31 00:35:55.233445807 +0000 UTC m=+2239.294220503" Oct 31 00:35:59.601431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.601399 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j"] Oct 31 00:35:59.604734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.604714 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:35:59.613481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.613455 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j"] Oct 31 00:35:59.671105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.671079 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrpm\" (UniqueName: \"kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j\" (UID: \"7e51c022-4169-4c3e-a781-333812b1dcfb\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:35:59.771696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.771669 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrpm\" (UniqueName: \"kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j\" (UID: \"7e51c022-4169-4c3e-a781-333812b1dcfb\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:35:59.784605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.784581 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrpm\" (UniqueName: \"kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j\" (UID: \"7e51c022-4169-4c3e-a781-333812b1dcfb\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:35:59.918253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:35:59.918195 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:36:00.194012 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.193955 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j"] Oct 31 00:36:00.196986 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:00.196957 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e51c022_4169_4c3e_a781_333812b1dcfb.slice/crio-83affdfbb29a2642b75c0d4271997c8ed9feaa8a760bff4ef37cf5586eaf2129 WatchSource:0}: Error finding container 83affdfbb29a2642b75c0d4271997c8ed9feaa8a760bff4ef37cf5586eaf2129: Status 404 returned error can't find the container with id 83affdfbb29a2642b75c0d4271997c8ed9feaa8a760bff4ef37cf5586eaf2129 Oct 31 00:36:00.221589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.221564 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" event={"ID":"7e51c022-4169-4c3e-a781-333812b1dcfb","Type":"ContainerStarted","Data":"83affdfbb29a2642b75c0d4271997c8ed9feaa8a760bff4ef37cf5586eaf2129"} Oct 31 00:36:00.416243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.416186 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:36:00.420060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.420038 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.445946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.445876 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:36:00.477315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477293 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477321 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477348 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477399 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477485 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477514 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477535 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477646 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477561 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477641 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477679 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdt6\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.477749 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.477719 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.478170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.478144 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.511168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511142 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:36:00.511168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511165 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:36:00.511326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511148 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:36:00.511326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511189 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:36:00.511326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511142 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:36:00.511326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511224 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:36:00.511326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.511243 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dlc5w\"" Oct 31 00:36:00.513946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.513927 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:36:00.514062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.513933 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:36:00.514062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.513995 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:36:00.584652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584629 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.584769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584657 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.584769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584679 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.584769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584697 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.584769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584713 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdt6\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584854 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584891 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584940 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.584961 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.585010 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.585033 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.585066 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.585376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.585352 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.586285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.586257 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.586976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.586951 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.588149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.588119 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.588242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.588199 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.588314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.588283 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.588406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.588375 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.588503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.588430 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.588902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.588886 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.589343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.589320 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.589417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.589402 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:00.597225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:00.597203 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdt6\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:01.228252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.228211 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" event={"ID":"7e51c022-4169-4c3e-a781-333812b1dcfb","Type":"ContainerStarted","Data":"43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a"} Oct 31 00:36:01.245236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.245186 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" podStartSLOduration=2.245173296 podStartE2EDuration="2.245173296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:01.244752867 +0000 UTC m=+2245.305527565" watchObservedRunningTime="2026-10-31 00:36:01.245173296 +0000 UTC m=+2245.305947994" Oct 31 00:36:01.275643 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.275619 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk"] Oct 31 00:36:01.275834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.275775 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" podUID="ddd9ff87-74f5-4006-8cf3-b451294222a8" containerName="thanos-querier" containerID="cri-o://c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3" gracePeriod=30 Oct 31 00:36:01.516659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.516636 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:36:01.592587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.592552 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz\" (UniqueName: \"kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz\") pod \"ddd9ff87-74f5-4006-8cf3-b451294222a8\" (UID: \"ddd9ff87-74f5-4006-8cf3-b451294222a8\") " Oct 31 00:36:01.594695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.594674 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz" pod "ddd9ff87-74f5-4006-8cf3-b451294222a8" (UID: "ddd9ff87-74f5-4006-8cf3-b451294222a8"). InnerVolumeSpecName "kube-api-access-ws8lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:01.693710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:01.693687 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8lz\" (UniqueName: \"kubernetes.io/projected/ddd9ff87-74f5-4006-8cf3-b451294222a8-kube-api-access-ws8lz\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:02.234477 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.234442 5073 generic.go:356] "Generic (PLEG): container finished" podID="ddd9ff87-74f5-4006-8cf3-b451294222a8" containerID="c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3" exitCode=0 Oct 31 00:36:02.234477 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.234477 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" Oct 31 00:36:02.234967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.234525 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" event={"ID":"ddd9ff87-74f5-4006-8cf3-b451294222a8","Type":"ContainerDied","Data":"c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3"} Oct 31 00:36:02.234967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.234560 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk" event={"ID":"ddd9ff87-74f5-4006-8cf3-b451294222a8","Type":"ContainerDied","Data":"33e92ec148447955edcd5d0e8dc8ca97e18f143b9ca0ed90e3f9a57ffd7c3810"} Oct 31 00:36:02.234967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.234579 5073 scope.go:122] "RemoveContainer" containerID="c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3" Oct 31 00:36:02.243210 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.243191 5073 scope.go:122] "RemoveContainer" containerID="c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3" Oct 31 00:36:02.243461 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:36:02.243442 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3\": container with ID starting with c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3 not found: ID does not exist" containerID="c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3" Oct 31 00:36:02.243544 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.243472 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3"} err="failed to get container status \"c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3\": rpc error: code = NotFound desc = could not find container \"c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3\": container with ID starting with c082ce8c5e68d2ac8ee3aa786b41a36b2a48364063abebb689b5e5813e4cf3b3 not found: ID does not exist" Oct 31 00:36:02.253920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.253901 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk"] Oct 31 00:36:02.260159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:02.260137 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-bcttk"] Oct 31 00:36:04.058570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.058541 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd9ff87-74f5-4006-8cf3-b451294222a8" path="/var/lib/kubelet/pods/ddd9ff87-74f5-4006-8cf3-b451294222a8/volumes" Oct 31 00:36:04.113814 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.113765 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.122203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.122181 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.214709 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.214674 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.214882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.214833 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.218361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.218340 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.310168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.310107 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.326563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.326539 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.341264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.341238 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:04.628701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:04.628571 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:36:04.632337 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:04.632305 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430594d8_210a_4871_80be_e8afcc0ce495.slice/crio-1bb3c80b78616f4e1ab008468a95947db7fa45c026e9154da497f51e3770063b WatchSource:0}: Error finding container 1bb3c80b78616f4e1ab008468a95947db7fa45c026e9154da497f51e3770063b: Status 404 returned error can't find the container with id 1bb3c80b78616f4e1ab008468a95947db7fa45c026e9154da497f51e3770063b Oct 31 00:36:05.250067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:05.250028 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerStarted","Data":"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e"} Oct 31 00:36:05.250470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:05.250093 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerStarted","Data":"1bb3c80b78616f4e1ab008468a95947db7fa45c026e9154da497f51e3770063b"} Oct 31 00:36:08.265114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:08.265079 5073 generic.go:356] "Generic (PLEG): container finished" podID="430594d8-210a-4871-80be-e8afcc0ce495" containerID="93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e" exitCode=0 Oct 31 00:36:08.265466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:08.265127 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerDied","Data":"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e"} Oct 31 00:36:09.272943 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:09.272907 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerStarted","Data":"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6"} Oct 31 00:36:09.273426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:09.272988 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerStarted","Data":"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b"} Oct 31 00:36:09.273426 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:09.273027 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerStarted","Data":"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6"} Oct 31 00:36:09.304915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:09.304863 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.218676217 podStartE2EDuration="9.304851857s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.08617564s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:09.303393104 +0000 UTC m=+2253.364167792" watchObservedRunningTime="2026-10-31 00:36:09.304851857 +0000 UTC m=+2253.365626552" Oct 31 00:36:09.341607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:09.341587 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:13.084199 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:36:13.084161 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:36:19.341438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:19.341404 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:19.346759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:19.346737 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:20.326531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:20.326498 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:36:33.451031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.450998 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z"] Oct 31 00:36:33.451530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.451510 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ddd9ff87-74f5-4006-8cf3-b451294222a8" containerName="thanos-querier" Oct 31 00:36:33.451669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.451533 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd9ff87-74f5-4006-8cf3-b451294222a8" containerName="thanos-querier" Oct 31 00:36:33.451669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.451651 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd9ff87-74f5-4006-8cf3-b451294222a8" containerName="thanos-querier" Oct 31 00:36:33.454841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.454819 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.467089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.467029 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z"] Oct 31 00:36:33.543376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.543345 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.543536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.543459 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7qs\" (UniqueName: \"kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.543536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.543495 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.644117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.644094 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7qs\" (UniqueName: \"kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.644279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.644125 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.644279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.644175 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.644885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.644863 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.646880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.646855 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.659261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.659232 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7qs\" (UniqueName: \"kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs\") pod \"data-science-collector-targetallocator-75d89d9c68-4cw2z\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:33.768313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:33.768291 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:34.046380 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.046351 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z"] Oct 31 00:36:34.049292 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:34.049267 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1477a0a0_65f9_4f74_9fe3_0642818902b4.slice/crio-bdfcf89e5e98d26159da5add0efe8e7cb48d43ce8f526575f550013f8f0bc71f WatchSource:0}: Error finding container bdfcf89e5e98d26159da5add0efe8e7cb48d43ce8f526575f550013f8f0bc71f: Status 404 returned error can't find the container with id bdfcf89e5e98d26159da5add0efe8e7cb48d43ce8f526575f550013f8f0bc71f Oct 31 00:36:34.051422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.051388 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:36:34.051528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.051450 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:36:34.387180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.387100 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" event={"ID":"1477a0a0-65f9-4f74-9fe3-0642818902b4","Type":"ContainerStarted","Data":"49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf"} Oct 31 00:36:34.387180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.387150 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" event={"ID":"1477a0a0-65f9-4f74-9fe3-0642818902b4","Type":"ContainerStarted","Data":"bdfcf89e5e98d26159da5add0efe8e7cb48d43ce8f526575f550013f8f0bc71f"} Oct 31 00:36:34.387364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.387239 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:34.388522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.388501 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:34.408715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.408672 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" podStartSLOduration=1.408659439 podStartE2EDuration="1.408659439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:34.407256674 +0000 UTC m=+2278.468031371" watchObservedRunningTime="2026-10-31 00:36:34.408659439 +0000 UTC m=+2278.469434163" Oct 31 00:36:34.456247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.456222 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98"] Oct 31 00:36:34.456602 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.456386 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" podUID="caeafbb4-c94c-4efc-a44d-f7ae28847d90" containerName="ta-container" containerID="cri-o://53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201" gracePeriod=30 Oct 31 00:36:34.704236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.704209 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:36:34.753094 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.753066 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal\" (UniqueName: \"kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal\") pod \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " Oct 31 00:36:34.753216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.753107 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert\") pod \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " Oct 31 00:36:34.753216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.753156 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v\" (UniqueName: \"kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v\") pod \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\" (UID: \"caeafbb4-c94c-4efc-a44d-f7ae28847d90\") " Oct 31 00:36:34.753477 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.753458 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal" pod "caeafbb4-c94c-4efc-a44d-f7ae28847d90" (UID: "caeafbb4-c94c-4efc-a44d-f7ae28847d90"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:34.755352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.755328 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert" pod "caeafbb4-c94c-4efc-a44d-f7ae28847d90" (UID: "caeafbb4-c94c-4efc-a44d-f7ae28847d90"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:34.755352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.755337 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v" pod "caeafbb4-c94c-4efc-a44d-f7ae28847d90" (UID: "caeafbb4-c94c-4efc-a44d-f7ae28847d90"). InnerVolumeSpecName "kube-api-access-wzh6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:34.854225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.854200 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/caeafbb4-c94c-4efc-a44d-f7ae28847d90-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.854225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.854222 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzh6v\" (UniqueName: \"kubernetes.io/projected/caeafbb4-c94c-4efc-a44d-f7ae28847d90-kube-api-access-wzh6v\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.854371 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:34.854234 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/caeafbb4-c94c-4efc-a44d-f7ae28847d90-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:35.392828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.392795 5073 generic.go:356] "Generic (PLEG): container finished" podID="caeafbb4-c94c-4efc-a44d-f7ae28847d90" containerID="53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201" exitCode=0 Oct 31 00:36:35.392992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.392833 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" Oct 31 00:36:35.392992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.392877 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" event={"ID":"caeafbb4-c94c-4efc-a44d-f7ae28847d90","Type":"ContainerDied","Data":"53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201"} Oct 31 00:36:35.392992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.392926 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98" event={"ID":"caeafbb4-c94c-4efc-a44d-f7ae28847d90","Type":"ContainerDied","Data":"4f115751a0d43fe19151cb080a5853566f56e32c9a288bd74a6d2031c5c39bfe"} Oct 31 00:36:35.392992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.392943 5073 scope.go:122] "RemoveContainer" containerID="53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201" Oct 31 00:36:35.402569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.402554 5073 scope.go:122] "RemoveContainer" containerID="53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201" Oct 31 00:36:35.402859 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:36:35.402841 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201\": container with ID starting with 53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201 not found: ID does not exist" containerID="53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201" Oct 31 00:36:35.402915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.402869 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201"} err="failed to get container status \"53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201\": rpc error: code = NotFound desc = could not find container \"53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201\": container with ID starting with 53e4a22d71557b245f1f828743f00a92cc1c013900a1331d05dc98dcfc59e201 not found: ID does not exist" Oct 31 00:36:35.414300 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.414278 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98"] Oct 31 00:36:35.418280 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:35.418261 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7dd4b5ccd7-6dd98"] Oct 31 00:36:36.059163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:36.059131 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caeafbb4-c94c-4efc-a44d-f7ae28847d90" path="/var/lib/kubelet/pods/caeafbb4-c94c-4efc-a44d-f7ae28847d90/volumes" Oct 31 00:36:39.712453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.712421 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02"] Oct 31 00:36:39.712889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.712836 5073 cpu_manager.go:414] "containerMap: removing container" podUID="caeafbb4-c94c-4efc-a44d-f7ae28847d90" containerName="ta-container" Oct 31 00:36:39.712889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.712849 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caeafbb4-c94c-4efc-a44d-f7ae28847d90" containerName="ta-container" Oct 31 00:36:39.712979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.712922 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="caeafbb4-c94c-4efc-a44d-f7ae28847d90" containerName="ta-container" Oct 31 00:36:39.717450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.717433 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" Oct 31 00:36:39.722682 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.722654 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02"] Oct 31 00:36:39.729498 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:39.729478 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" Oct 31 00:36:40.002351 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:40.002315 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02"] Oct 31 00:36:40.005365 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:40.005335 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod733cf816_ecc9_4036_905b_1e204d39fbf9.slice/crio-a630d9d0f13561626a6694479e12ff43189a77412eb33d063eafec98d81b2f11 WatchSource:0}: Error finding container a630d9d0f13561626a6694479e12ff43189a77412eb33d063eafec98d81b2f11: Status 404 returned error can't find the container with id a630d9d0f13561626a6694479e12ff43189a77412eb33d063eafec98d81b2f11 Oct 31 00:36:40.418699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:40.418611 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" event={"ID":"733cf816-ecc9-4036-905b-1e204d39fbf9","Type":"ContainerStarted","Data":"4fe063f34a499c6a1851c9dcc0c33d35c459a3edf226ac1b68196ee028c5aa79"} Oct 31 00:36:40.418699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:40.418655 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" event={"ID":"733cf816-ecc9-4036-905b-1e204d39fbf9","Type":"ContainerStarted","Data":"a630d9d0f13561626a6694479e12ff43189a77412eb33d063eafec98d81b2f11"} Oct 31 00:36:40.434348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:40.434299 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" podStartSLOduration=1.434283974 podStartE2EDuration="1.434283974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:40.433759518 +0000 UTC m=+2284.494534195" watchObservedRunningTime="2026-10-31 00:36:40.434283974 +0000 UTC m=+2284.495058670" Oct 31 00:36:43.085961 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:36:43.085926 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:36:45.444464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:45.444385 5073 generic.go:356] "Generic (PLEG): container finished" podID="733cf816-ecc9-4036-905b-1e204d39fbf9" containerID="4fe063f34a499c6a1851c9dcc0c33d35c459a3edf226ac1b68196ee028c5aa79" exitCode=0 Oct 31 00:36:45.444930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:45.444463 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" event={"ID":"733cf816-ecc9-4036-905b-1e204d39fbf9","Type":"ContainerDied","Data":"4fe063f34a499c6a1851c9dcc0c33d35c459a3edf226ac1b68196ee028c5aa79"} Oct 31 00:36:46.585504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:46.585479 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" Oct 31 00:36:47.454567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.454540 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" Oct 31 00:36:47.454744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.454539 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02" event={"ID":"733cf816-ecc9-4036-905b-1e204d39fbf9","Type":"ContainerDied","Data":"a630d9d0f13561626a6694479e12ff43189a77412eb33d063eafec98d81b2f11"} Oct 31 00:36:47.454744 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.454662 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a630d9d0f13561626a6694479e12ff43189a77412eb33d063eafec98d81b2f11" Oct 31 00:36:47.823033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.822996 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02"] Oct 31 00:36:47.828699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.828669 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-perses-18e3762a11271b02"] Oct 31 00:36:47.885667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.885637 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa"] Oct 31 00:36:47.886109 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.886091 5073 cpu_manager.go:414] "containerMap: removing container" podUID="733cf816-ecc9-4036-905b-1e204d39fbf9" containerName="curl" Oct 31 00:36:47.886225 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.886113 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733cf816-ecc9-4036-905b-1e204d39fbf9" containerName="curl" Oct 31 00:36:47.886322 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.886225 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="733cf816-ecc9-4036-905b-1e204d39fbf9" containerName="curl" Oct 31 00:36:47.889308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.889290 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa" Oct 31 00:36:47.896549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.896521 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa"] Oct 31 00:36:47.901204 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:47.901185 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa" Oct 31 00:36:48.059036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:48.059003 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733cf816-ecc9-4036-905b-1e204d39fbf9" path="/var/lib/kubelet/pods/733cf816-ecc9-4036-905b-1e204d39fbf9/volumes" Oct 31 00:36:48.174725 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:48.174658 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa"] Oct 31 00:36:48.177907 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:48.177880 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae856fe0_518d_4fb0_8341_ffd60892d155.slice/crio-bc412d6fe08542262a33aa1510d5ab9400aedc59f275e36d48485539624b34cc WatchSource:0}: Error finding container bc412d6fe08542262a33aa1510d5ab9400aedc59f275e36d48485539624b34cc: Status 404 returned error can't find the container with id bc412d6fe08542262a33aa1510d5ab9400aedc59f275e36d48485539624b34cc Oct 31 00:36:48.460965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:48.460903 5073 generic.go:356] "Generic (PLEG): container finished" podID="ae856fe0-518d-4fb0-8341-ffd60892d155" containerID="823ba0f705252a4c6898f7736c9e2330e1e06d7910bccf7b25d0fc00d1d8ca50" exitCode=0 Oct 31 00:36:48.461078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:48.460982 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa" event={"ID":"ae856fe0-518d-4fb0-8341-ffd60892d155","Type":"ContainerDied","Data":"823ba0f705252a4c6898f7736c9e2330e1e06d7910bccf7b25d0fc00d1d8ca50"} Oct 31 00:36:48.461078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:48.461008 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa" event={"ID":"ae856fe0-518d-4fb0-8341-ffd60892d155","Type":"ContainerStarted","Data":"bc412d6fe08542262a33aa1510d5ab9400aedc59f275e36d48485539624b34cc"} Oct 31 00:36:49.600063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.600038 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa" Oct 31 00:36:49.936325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.936296 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa"] Oct 31 00:36:49.939960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.939931 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa"] Oct 31 00:36:49.996342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.996318 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8"] Oct 31 00:36:49.996699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.996687 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ae856fe0-518d-4fb0-8341-ffd60892d155" containerName="curl" Oct 31 00:36:49.996740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.996702 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae856fe0-518d-4fb0-8341-ffd60892d155" containerName="curl" Oct 31 00:36:49.996777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.996764 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae856fe0-518d-4fb0-8341-ffd60892d155" containerName="curl" Oct 31 00:36:49.999649 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:49.999634 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8" Oct 31 00:36:50.006983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.006946 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8"] Oct 31 00:36:50.011776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.011753 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8" Oct 31 00:36:50.061096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.061064 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae856fe0-518d-4fb0-8341-ffd60892d155" path="/var/lib/kubelet/pods/ae856fe0-518d-4fb0-8341-ffd60892d155/volumes" Oct 31 00:36:50.287390 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.287361 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8"] Oct 31 00:36:50.290323 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:50.290282 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c19b5f7_3db0_4e14_9c68_186bd5657d75.slice/crio-373fa2b549fecea62312f55feb4ec5bf7c1b366539f1a24c707152dc7b2284f5 WatchSource:0}: Error finding container 373fa2b549fecea62312f55feb4ec5bf7c1b366539f1a24c707152dc7b2284f5: Status 404 returned error can't find the container with id 373fa2b549fecea62312f55feb4ec5bf7c1b366539f1a24c707152dc7b2284f5 Oct 31 00:36:50.476832 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.476805 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-korrel8r-18e3762bf8b1d3aa" Oct 31 00:36:50.476999 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.476832 5073 scope.go:122] "RemoveContainer" containerID="823ba0f705252a4c6898f7736c9e2330e1e06d7910bccf7b25d0fc00d1d8ca50" Oct 31 00:36:50.478382 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.478355 5073 generic.go:356] "Generic (PLEG): container finished" podID="6c19b5f7-3db0-4e14-9c68-186bd5657d75" containerID="a2f930bd3cd5aa9ac8af5c0e34dd535000d1113c430697b6c1ba378f32709461" exitCode=0 Oct 31 00:36:50.478500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.478436 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8" event={"ID":"6c19b5f7-3db0-4e14-9c68-186bd5657d75","Type":"ContainerDied","Data":"a2f930bd3cd5aa9ac8af5c0e34dd535000d1113c430697b6c1ba378f32709461"} Oct 31 00:36:50.478500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:50.478473 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8" event={"ID":"6c19b5f7-3db0-4e14-9c68-186bd5657d75","Type":"ContainerStarted","Data":"373fa2b549fecea62312f55feb4ec5bf7c1b366539f1a24c707152dc7b2284f5"} Oct 31 00:36:51.617349 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:51.617322 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8" Oct 31 00:36:52.046725 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.046689 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8"] Oct 31 00:36:52.048645 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.048618 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8"] Oct 31 00:36:52.059109 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.059079 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c19b5f7-3db0-4e14-9c68-186bd5657d75" path="/var/lib/kubelet/pods/6c19b5f7-3db0-4e14-9c68-186bd5657d75/volumes" Oct 31 00:36:52.121042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.121014 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573"] Oct 31 00:36:52.121382 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.121370 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6c19b5f7-3db0-4e14-9c68-186bd5657d75" containerName="curl" Oct 31 00:36:52.121430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.121383 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c19b5f7-3db0-4e14-9c68-186bd5657d75" containerName="curl" Oct 31 00:36:52.121465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.121459 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c19b5f7-3db0-4e14-9c68-186bd5657d75" containerName="curl" Oct 31 00:36:52.124415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.124398 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" Oct 31 00:36:52.129364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.129343 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"thanos-ingress-probe-18e3762cf33be874\"/\"default-dockercfg-xjm82\"" Oct 31 00:36:52.133918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.133896 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573"] Oct 31 00:36:52.136076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.136057 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" Oct 31 00:36:52.413039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.412974 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573"] Oct 31 00:36:52.415984 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:52.415959 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded1c9ca7_b552_4e8e_a527_d2b0c942c966.slice/crio-da6b4d6eacfe87f0d6f1196cda5986a2e10a3bf92a1cd4a482e309c2749e248a WatchSource:0}: Error finding container da6b4d6eacfe87f0d6f1196cda5986a2e10a3bf92a1cd4a482e309c2749e248a: Status 404 returned error can't find the container with id da6b4d6eacfe87f0d6f1196cda5986a2e10a3bf92a1cd4a482e309c2749e248a Oct 31 00:36:52.490968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.490937 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" event={"ID":"ed1c9ca7-b552-4e8e-a527-d2b0c942c966","Type":"ContainerStarted","Data":"28c0fc704324759991f634859f98138ee1cb53849ce788d2c59c83d932ddd8e4"} Oct 31 00:36:52.491075 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.490979 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" event={"ID":"ed1c9ca7-b552-4e8e-a527-d2b0c942c966","Type":"ContainerStarted","Data":"da6b4d6eacfe87f0d6f1196cda5986a2e10a3bf92a1cd4a482e309c2749e248a"} Oct 31 00:36:52.492277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.492255 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-authenticated-proxy-18e3762c76930ab8" Oct 31 00:36:52.492364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.492263 5073 scope.go:122] "RemoveContainer" containerID="a2f930bd3cd5aa9ac8af5c0e34dd535000d1113c430697b6c1ba378f32709461" Oct 31 00:36:52.524040 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:52.523999 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" podStartSLOduration=0.523986391 podStartE2EDuration="523.986391ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:52.521968081 +0000 UTC m=+2296.582742789" watchObservedRunningTime="2026-10-31 00:36:52.523986391 +0000 UTC m=+2296.584761086" Oct 31 00:36:53.616752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.616718 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w"] Oct 31 00:36:53.621306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.621289 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.630411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.630384 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w"] Oct 31 00:36:53.707791 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.707763 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.707923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.707881 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.707968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.707920 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqzb\" (UniqueName: \"kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.809024 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.808988 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.809183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.809044 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.809183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.809071 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqzb\" (UniqueName: \"kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.809603 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.809581 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.811800 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.811760 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.823404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.823383 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqzb\" (UniqueName: \"kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb\") pod \"data-science-collector-targetallocator-79ff84bf85-l6g4w\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:53.933812 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:53.933723 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:54.207272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.207198 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w"] Oct 31 00:36:54.210397 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:36:54.210368 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e0771bb_350f_42ae_9a7c_fdc2f53c6f1d.slice/crio-3cb3ccfc38155ad8cb567624221b571d8faea281c5e73d7288d0a3fe288a39b2 WatchSource:0}: Error finding container 3cb3ccfc38155ad8cb567624221b571d8faea281c5e73d7288d0a3fe288a39b2: Status 404 returned error can't find the container with id 3cb3ccfc38155ad8cb567624221b571d8faea281c5e73d7288d0a3fe288a39b2 Oct 31 00:36:54.213015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.212975 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:36:54.213087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.213056 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:36:54.504752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.504711 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" event={"ID":"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d","Type":"ContainerStarted","Data":"7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad"} Oct 31 00:36:54.504957 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.504854 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:54.504957 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.504889 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" event={"ID":"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d","Type":"ContainerStarted","Data":"3cb3ccfc38155ad8cb567624221b571d8faea281c5e73d7288d0a3fe288a39b2"} Oct 31 00:36:54.506588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.506567 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:36:54.524230 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.524188 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" podStartSLOduration=1.5241753390000001 podStartE2EDuration="1.524175339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:54.522592306 +0000 UTC m=+2298.583367011" watchObservedRunningTime="2026-10-31 00:36:54.524175339 +0000 UTC m=+2298.584950033" Oct 31 00:36:54.570433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.570404 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z"] Oct 31 00:36:54.570612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.570587 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" podUID="1477a0a0-65f9-4f74-9fe3-0642818902b4" containerName="ta-container" containerID="cri-o://49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf" gracePeriod=30 Oct 31 00:36:54.802840 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.802817 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:54.918880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.918850 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs\" (UniqueName: \"kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs\") pod \"1477a0a0-65f9-4f74-9fe3-0642818902b4\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " Oct 31 00:36:54.919002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.918964 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal\" (UniqueName: \"kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal\") pod \"1477a0a0-65f9-4f74-9fe3-0642818902b4\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " Oct 31 00:36:54.919002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.918993 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert\") pod \"1477a0a0-65f9-4f74-9fe3-0642818902b4\" (UID: \"1477a0a0-65f9-4f74-9fe3-0642818902b4\") " Oct 31 00:36:54.919306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.919274 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal" pod "1477a0a0-65f9-4f74-9fe3-0642818902b4" (UID: "1477a0a0-65f9-4f74-9fe3-0642818902b4"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:54.921153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.921130 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs" pod "1477a0a0-65f9-4f74-9fe3-0642818902b4" (UID: "1477a0a0-65f9-4f74-9fe3-0642818902b4"). InnerVolumeSpecName "kube-api-access-zw7qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:54.921237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:54.921129 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert" pod "1477a0a0-65f9-4f74-9fe3-0642818902b4" (UID: "1477a0a0-65f9-4f74-9fe3-0642818902b4"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:55.019533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.019508 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/1477a0a0-65f9-4f74-9fe3-0642818902b4-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:55.019533 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.019531 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/1477a0a0-65f9-4f74-9fe3-0642818902b4-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:55.019659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.019542 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw7qs\" (UniqueName: \"kubernetes.io/projected/1477a0a0-65f9-4f74-9fe3-0642818902b4-kube-api-access-zw7qs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:55.511103 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.511076 5073 generic.go:356] "Generic (PLEG): container finished" podID="1477a0a0-65f9-4f74-9fe3-0642818902b4" containerID="49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf" exitCode=0 Oct 31 00:36:55.511293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.511129 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" Oct 31 00:36:55.511293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.511162 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" event={"ID":"1477a0a0-65f9-4f74-9fe3-0642818902b4","Type":"ContainerDied","Data":"49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf"} Oct 31 00:36:55.511293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.511205 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z" event={"ID":"1477a0a0-65f9-4f74-9fe3-0642818902b4","Type":"ContainerDied","Data":"bdfcf89e5e98d26159da5add0efe8e7cb48d43ce8f526575f550013f8f0bc71f"} Oct 31 00:36:55.511293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.511222 5073 scope.go:122] "RemoveContainer" containerID="49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf" Oct 31 00:36:55.520404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.520387 5073 scope.go:122] "RemoveContainer" containerID="49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf" Oct 31 00:36:55.520653 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:36:55.520633 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf\": container with ID starting with 49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf not found: ID does not exist" containerID="49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf" Oct 31 00:36:55.520716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.520666 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf"} err="failed to get container status \"49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf\": rpc error: code = NotFound desc = could not find container \"49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf\": container with ID starting with 49818c05153ecd98f77a727fa1c2137da96c82d87dbf24905aaabba5842741bf not found: ID does not exist" Oct 31 00:36:55.532719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.532699 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z"] Oct 31 00:36:55.536597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:55.536578 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-75d89d9c68-4cw2z"] Oct 31 00:36:56.058735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:56.058706 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1477a0a0-65f9-4f74-9fe3-0642818902b4" path="/var/lib/kubelet/pods/1477a0a0-65f9-4f74-9fe3-0642818902b4/volumes" Oct 31 00:36:57.523842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:57.523817 5073 generic.go:356] "Generic (PLEG): container finished" podID="ed1c9ca7-b552-4e8e-a527-d2b0c942c966" containerID="28c0fc704324759991f634859f98138ee1cb53849ce788d2c59c83d932ddd8e4" exitCode=0 Oct 31 00:36:57.524142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:57.523859 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" event={"ID":"ed1c9ca7-b552-4e8e-a527-d2b0c942c966","Type":"ContainerDied","Data":"28c0fc704324759991f634859f98138ee1cb53849ce788d2c59c83d932ddd8e4"} Oct 31 00:36:58.662673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:58.662650 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" Oct 31 00:36:59.534042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:59.534009 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" event={"ID":"ed1c9ca7-b552-4e8e-a527-d2b0c942c966","Type":"ContainerDied","Data":"da6b4d6eacfe87f0d6f1196cda5986a2e10a3bf92a1cd4a482e309c2749e248a"} Oct 31 00:36:59.534042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:59.534039 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573" Oct 31 00:36:59.534252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:36:59.534046 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6b4d6eacfe87f0d6f1196cda5986a2e10a3bf92a1cd4a482e309c2749e248a" Oct 31 00:37:00.230855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:00.230822 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573"] Oct 31 00:37:00.235040 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:00.235014 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["thanos-ingress-probe-18e3762cf33be874/thanos-ingress-untrusted-18e3762cf4630573"] Oct 31 00:37:02.057892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:02.057860 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1c9ca7-b552-4e8e-a527-d2b0c942c966" path="/var/lib/kubelet/pods/ed1c9ca7-b552-4e8e-a527-d2b0c942c966/volumes" Oct 31 00:37:10.543420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.543365 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6"] Oct 31 00:37:10.544095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.544032 5073 cpu_manager.go:414] "containerMap: removing container" podUID="1477a0a0-65f9-4f74-9fe3-0642818902b4" containerName="ta-container" Oct 31 00:37:10.544095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.544055 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1477a0a0-65f9-4f74-9fe3-0642818902b4" containerName="ta-container" Oct 31 00:37:10.544095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.544067 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ed1c9ca7-b552-4e8e-a527-d2b0c942c966" containerName="curl" Oct 31 00:37:10.544095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.544077 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1c9ca7-b552-4e8e-a527-d2b0c942c966" containerName="curl" Oct 31 00:37:10.544316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.544164 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="1477a0a0-65f9-4f74-9fe3-0642818902b4" containerName="ta-container" Oct 31 00:37:10.544316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.544174 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1c9ca7-b552-4e8e-a527-d2b0c942c966" containerName="curl" Oct 31 00:37:10.548566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.548544 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.559891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.559860 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6"] Oct 31 00:37:10.639148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.639114 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.639303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.639157 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p84h\" (UniqueName: \"kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.639303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.639184 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.739671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.739639 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.739828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.739684 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p84h\" (UniqueName: \"kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.739828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.739706 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.740366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.740337 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.742280 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.742257 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.752013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.751983 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p84h\" (UniqueName: \"kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h\") pod \"data-science-collector-targetallocator-666b69549-8qhp6\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:10.860841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:10.860766 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:11.139427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.139347 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6"] Oct 31 00:37:11.144020 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:37:11.143982 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04a2e696_4d5d_4a8c_86a7_b355b0267363.slice/crio-bdadb73aa16133ec7b7813b7b47e44c144f711eac24166a81685b4c1361bb6f9 WatchSource:0}: Error finding container bdadb73aa16133ec7b7813b7b47e44c144f711eac24166a81685b4c1361bb6f9: Status 404 returned error can't find the container with id bdadb73aa16133ec7b7813b7b47e44c144f711eac24166a81685b4c1361bb6f9 Oct 31 00:37:11.146102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.146061 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:11.146198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.146132 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:11.592513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.592475 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" event={"ID":"04a2e696-4d5d-4a8c-86a7-b355b0267363","Type":"ContainerStarted","Data":"82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7"} Oct 31 00:37:11.593003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.592522 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" event={"ID":"04a2e696-4d5d-4a8c-86a7-b355b0267363","Type":"ContainerStarted","Data":"bdadb73aa16133ec7b7813b7b47e44c144f711eac24166a81685b4c1361bb6f9"} Oct 31 00:37:11.593003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.592669 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:11.594173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.594149 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:11.633127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.633085 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" podStartSLOduration=1.633071695 podStartE2EDuration="1.633071695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:11.612646411 +0000 UTC m=+2315.673421107" watchObservedRunningTime="2026-10-31 00:37:11.633071695 +0000 UTC m=+2315.693846390" Oct 31 00:37:11.662712 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.662685 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w"] Oct 31 00:37:11.662914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.662889 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" podUID="3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" containerName="ta-container" containerID="cri-o://7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad" gracePeriod=30 Oct 31 00:37:11.902510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:11.902488 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:37:12.051117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.051090 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb\" (UniqueName: \"kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb\") pod \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " Oct 31 00:37:12.051266 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.051138 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert\") pod \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " Oct 31 00:37:12.051266 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.051172 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal\") pod \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\" (UID: \"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d\") " Oct 31 00:37:12.051578 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.051551 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal" pod "3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" (UID: "3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:12.053369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.053344 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb" pod "3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" (UID: "3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d"). InnerVolumeSpecName "kube-api-access-ktqzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:12.053478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.053439 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert" pod "3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" (UID: "3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:12.151993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.151937 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqzb\" (UniqueName: \"kubernetes.io/projected/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-kube-api-access-ktqzb\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.151993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.151959 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.151993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.151969 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.598848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.598818 5073 generic.go:356] "Generic (PLEG): container finished" podID="3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" containerID="7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad" exitCode=0 Oct 31 00:37:12.599312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.598870 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" Oct 31 00:37:12.599312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.598903 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" event={"ID":"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d","Type":"ContainerDied","Data":"7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad"} Oct 31 00:37:12.599312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.598944 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w" event={"ID":"3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d","Type":"ContainerDied","Data":"3cb3ccfc38155ad8cb567624221b571d8faea281c5e73d7288d0a3fe288a39b2"} Oct 31 00:37:12.599312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.598962 5073 scope.go:122] "RemoveContainer" containerID="7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad" Oct 31 00:37:12.607481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.607462 5073 scope.go:122] "RemoveContainer" containerID="7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad" Oct 31 00:37:12.607740 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:12.607722 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad\": container with ID starting with 7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad not found: ID does not exist" containerID="7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad" Oct 31 00:37:12.607808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.607753 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad"} err="failed to get container status \"7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad\": rpc error: code = NotFound desc = could not find container \"7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad\": container with ID starting with 7290f8e99c219862b54668c8793d342c865988bcd5cd34adb77c4cf78f9f45ad not found: ID does not exist" Oct 31 00:37:12.615652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.615626 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w"] Oct 31 00:37:12.620958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:12.620933 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79ff84bf85-l6g4w"] Oct 31 00:37:13.085739 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:13.085700 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:37:13.130318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.130284 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:13.130808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.130766 5073 cpu_manager.go:414] "containerMap: removing container" podUID="3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" containerName="ta-container" Oct 31 00:37:13.130808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.130800 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" containerName="ta-container" Oct 31 00:37:13.130912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.130872 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" containerName="ta-container" Oct 31 00:37:13.134513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.134498 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.139592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.139567 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:13.142015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.141995 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:13.143296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.143273 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:37:13.145365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.145339 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:13.240119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.240091 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:13.240119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.240115 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:13.240304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.240114 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:13.260469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.260435 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.260555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.260481 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.260619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.260584 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.260692 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.260674 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.260752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.260709 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.260842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.260827 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27vh\" (UniqueName: \"kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361106 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361153 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361183 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361231 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27vh\" (UniqueName: \"kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361260 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361294 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.361887 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.361859 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.363924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.363900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.363924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.363916 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.364308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.364290 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.364349 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.364295 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.373607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.373590 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27vh\" (UniqueName: \"kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh\") pod \"data-science-collector-collector-1\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.447692 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.447670 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.722119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.722052 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:13.725113 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:37:13.725078 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c93e962_4289_4b97_9b91_6a283713d96d.slice/crio-7a908cb956b6ff8363995c616997b6d414f7bb156b08f6090d2d539d595a07e2 WatchSource:0}: Error finding container 7a908cb956b6ff8363995c616997b6d414f7bb156b08f6090d2d539d595a07e2: Status 404 returned error can't find the container with id 7a908cb956b6ff8363995c616997b6d414f7bb156b08f6090d2d539d595a07e2 Oct 31 00:37:13.727159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.727131 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:13.727219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:13.727186 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:14.059352 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:14.059320 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d" path="/var/lib/kubelet/pods/3e0771bb-350f-42ae-9a7c-fdc2f53c6f1d/volumes" Oct 31 00:37:14.611491 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:14.611451 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7c93e962-4289-4b97-9b91-6a283713d96d","Type":"ContainerStarted","Data":"a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7"} Oct 31 00:37:14.611666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:14.611511 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7c93e962-4289-4b97-9b91-6a283713d96d","Type":"ContainerStarted","Data":"7a908cb956b6ff8363995c616997b6d414f7bb156b08f6090d2d539d595a07e2"} Oct 31 00:37:14.632922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:14.632867 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6328507989999999 podStartE2EDuration="1.632850799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:14.631515439 +0000 UTC m=+2318.692290135" watchObservedRunningTime="2026-10-31 00:37:14.632850799 +0000 UTC m=+2318.693625494" Oct 31 00:37:22.588488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.588455 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:22.589001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.588624 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7c93e962-4289-4b97-9b91-6a283713d96d" containerName="otc-container" containerID="cri-o://a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7" gracePeriod=30 Oct 31 00:37:22.687989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.687951 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d"] Oct 31 00:37:22.691737 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.691710 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.701754 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.701702 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d"] Oct 31 00:37:22.733113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.732847 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g569\" (UniqueName: \"kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.733113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.732928 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.733113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.732963 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.831141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.831114 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:22.834044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834022 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs\") pod \"7c93e962-4289-4b97-9b91-6a283713d96d\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " Oct 31 00:37:22.834113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834078 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh\" (UniqueName: \"kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh\") pod \"7c93e962-4289-4b97-9b91-6a283713d96d\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " Oct 31 00:37:22.834113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834101 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal\") pod \"7c93e962-4289-4b97-9b91-6a283713d96d\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " Oct 31 00:37:22.834185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834138 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs\") pod \"7c93e962-4289-4b97-9b91-6a283713d96d\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " Oct 31 00:37:22.834219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834202 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert\") pod \"7c93e962-4289-4b97-9b91-6a283713d96d\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " Oct 31 00:37:22.834267 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834238 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs\") pod \"7c93e962-4289-4b97-9b91-6a283713d96d\" (UID: \"7c93e962-4289-4b97-9b91-6a283713d96d\") " Oct 31 00:37:22.834437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834412 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.834541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834497 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal" pod "7c93e962-4289-4b97-9b91-6a283713d96d" (UID: "7c93e962-4289-4b97-9b91-6a283713d96d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:22.834541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834532 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g569\" (UniqueName: \"kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.834834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834613 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.834834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.834694 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7c93e962-4289-4b97-9b91-6a283713d96d-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.835369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.835345 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.836993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.836968 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs" pod "7c93e962-4289-4b97-9b91-6a283713d96d" (UID: "7c93e962-4289-4b97-9b91-6a283713d96d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:22.837088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.836974 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh" pod "7c93e962-4289-4b97-9b91-6a283713d96d" (UID: "7c93e962-4289-4b97-9b91-6a283713d96d"). InnerVolumeSpecName "kube-api-access-t27vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:22.837088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.837078 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs" pod "7c93e962-4289-4b97-9b91-6a283713d96d" (UID: "7c93e962-4289-4b97-9b91-6a283713d96d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:22.837206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.837174 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert" pod "7c93e962-4289-4b97-9b91-6a283713d96d" (UID: "7c93e962-4289-4b97-9b91-6a283713d96d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:22.837206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.837182 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs" pod "7c93e962-4289-4b97-9b91-6a283713d96d" (UID: "7c93e962-4289-4b97-9b91-6a283713d96d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:22.837543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.837524 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.847162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.847143 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g569\" (UniqueName: \"kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569\") pod \"data-science-collector-targetallocator-5b9cb85c54-xd89d\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:22.935769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.935732 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.935769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.935764 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.935995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.935830 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.935995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.935844 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t27vh\" (UniqueName: \"kubernetes.io/projected/7c93e962-4289-4b97-9b91-6a283713d96d-kube-api-access-t27vh\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.935995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:22.935858 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c93e962-4289-4b97-9b91-6a283713d96d-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:23.007416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.007383 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:23.304348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.304321 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d"] Oct 31 00:37:23.306836 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:37:23.306813 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c41f56e_3509_4e95_8849_02ba05415a1f.slice/crio-62d6c3604eb032f749cfdd7ce2e9359dd3814c18bfd16d4545d8879a12f4d6c5 WatchSource:0}: Error finding container 62d6c3604eb032f749cfdd7ce2e9359dd3814c18bfd16d4545d8879a12f4d6c5: Status 404 returned error can't find the container with id 62d6c3604eb032f749cfdd7ce2e9359dd3814c18bfd16d4545d8879a12f4d6c5 Oct 31 00:37:23.308766 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.308733 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:23.308848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.308812 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:23.655451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.655380 5073 generic.go:356] "Generic (PLEG): container finished" podID="7c93e962-4289-4b97-9b91-6a283713d96d" containerID="a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7" exitCode=0 Oct 31 00:37:23.655451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.655443 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.655991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.655470 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7c93e962-4289-4b97-9b91-6a283713d96d","Type":"ContainerDied","Data":"a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7"} Oct 31 00:37:23.655991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.655528 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7c93e962-4289-4b97-9b91-6a283713d96d","Type":"ContainerDied","Data":"7a908cb956b6ff8363995c616997b6d414f7bb156b08f6090d2d539d595a07e2"} Oct 31 00:37:23.655991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.655559 5073 scope.go:122] "RemoveContainer" containerID="a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7" Oct 31 00:37:23.657251 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.657228 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" event={"ID":"9c41f56e-3509-4e95-8849-02ba05415a1f","Type":"ContainerStarted","Data":"5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42"} Oct 31 00:37:23.657381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.657271 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" event={"ID":"9c41f56e-3509-4e95-8849-02ba05415a1f","Type":"ContainerStarted","Data":"62d6c3604eb032f749cfdd7ce2e9359dd3814c18bfd16d4545d8879a12f4d6c5"} Oct 31 00:37:23.657458 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.657377 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:23.659192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.659170 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:23.664770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.664755 5073 scope.go:122] "RemoveContainer" containerID="a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7" Oct 31 00:37:23.665039 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:23.665021 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7\": container with ID starting with a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7 not found: ID does not exist" containerID="a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7" Oct 31 00:37:23.665090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.665047 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7"} err="failed to get container status \"a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7\": rpc error: code = NotFound desc = could not find container \"a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7\": container with ID starting with a7787d4ad64285854f4aabdcd2883c24745a4f9a9c04edd6c430dc13c73d58e7 not found: ID does not exist" Oct 31 00:37:23.692664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.692620 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" podStartSLOduration=1.692605919 podStartE2EDuration="1.692605919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:23.691727561 +0000 UTC m=+2327.752502268" watchObservedRunningTime="2026-10-31 00:37:23.692605919 +0000 UTC m=+2327.753380617" Oct 31 00:37:23.708487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.708464 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:23.712451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.712418 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:23.723336 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.723313 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6"] Oct 31 00:37:23.723466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.723448 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" podUID="04a2e696-4d5d-4a8c-86a7-b355b0267363" containerName="ta-container" containerID="cri-o://82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7" gracePeriod=30 Oct 31 00:37:23.740738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.740712 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:23.741319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.741253 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7c93e962-4289-4b97-9b91-6a283713d96d" containerName="otc-container" Oct 31 00:37:23.741319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.741271 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c93e962-4289-4b97-9b91-6a283713d96d" containerName="otc-container" Oct 31 00:37:23.741471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.741358 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c93e962-4289-4b97-9b91-6a283713d96d" containerName="otc-container" Oct 31 00:37:23.744752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.744733 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.750420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.750403 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:23.752716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.752691 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:23.752716 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.752702 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:23.753049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.752932 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:23.753600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.753575 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:23.753929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.753907 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:37:23.755262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.755240 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:23.844041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.844019 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.844157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.844050 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.844157 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.844068 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfm7\" (UniqueName: \"kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.844249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.844219 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.844299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.844254 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.844299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.844273 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.945610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.945543 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.945610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.945583 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.945610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.945602 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.945905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.945647 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.945905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.945668 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.945905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.945685 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfm7\" (UniqueName: \"kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.946488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.946459 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.948453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.948429 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.948563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.948542 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.948695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.948676 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.949028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.949011 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.958405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.958382 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfm7\" (UniqueName: \"kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7\") pod \"data-science-collector-collector-1\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:23.976991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:23.976973 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:24.046128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.046099 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert\") pod \"04a2e696-4d5d-4a8c-86a7-b355b0267363\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " Oct 31 00:37:24.046262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.046136 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal\" (UniqueName: \"kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal\") pod \"04a2e696-4d5d-4a8c-86a7-b355b0267363\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " Oct 31 00:37:24.046262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.046170 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h\" (UniqueName: \"kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h\") pod \"04a2e696-4d5d-4a8c-86a7-b355b0267363\" (UID: \"04a2e696-4d5d-4a8c-86a7-b355b0267363\") " Oct 31 00:37:24.046465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.046444 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal" pod "04a2e696-4d5d-4a8c-86a7-b355b0267363" (UID: "04a2e696-4d5d-4a8c-86a7-b355b0267363"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:24.048318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.048299 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h" pod "04a2e696-4d5d-4a8c-86a7-b355b0267363" (UID: "04a2e696-4d5d-4a8c-86a7-b355b0267363"). InnerVolumeSpecName "kube-api-access-4p84h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:24.048318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.048310 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert" pod "04a2e696-4d5d-4a8c-86a7-b355b0267363" (UID: "04a2e696-4d5d-4a8c-86a7-b355b0267363"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:24.058119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.058097 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c93e962-4289-4b97-9b91-6a283713d96d" path="/var/lib/kubelet/pods/7c93e962-4289-4b97-9b91-6a283713d96d/volumes" Oct 31 00:37:24.104159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.104141 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:24.147503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.147469 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/04a2e696-4d5d-4a8c-86a7-b355b0267363-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:24.147503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.147496 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/04a2e696-4d5d-4a8c-86a7-b355b0267363-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:24.147503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.147507 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p84h\" (UniqueName: \"kubernetes.io/projected/04a2e696-4d5d-4a8c-86a7-b355b0267363-kube-api-access-4p84h\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:24.396586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.396416 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:24.399138 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:37:24.399110 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07d9dd6_e6a7_4c5e_a466_9a16d3b95eb0.slice/crio-970b96b56487ab44e96e4ce9e2e95d3b420115494147d3ed14132e27849b961c WatchSource:0}: Error finding container 970b96b56487ab44e96e4ce9e2e95d3b420115494147d3ed14132e27849b961c: Status 404 returned error can't find the container with id 970b96b56487ab44e96e4ce9e2e95d3b420115494147d3ed14132e27849b961c Oct 31 00:37:24.401346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.401311 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:24.401443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.401385 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:24.664198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.664113 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0","Type":"ContainerStarted","Data":"28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb"} Oct 31 00:37:24.664198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.664173 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0","Type":"ContainerStarted","Data":"970b96b56487ab44e96e4ce9e2e95d3b420115494147d3ed14132e27849b961c"} Oct 31 00:37:24.665452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.665428 5073 generic.go:356] "Generic (PLEG): container finished" podID="04a2e696-4d5d-4a8c-86a7-b355b0267363" containerID="82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7" exitCode=0 Oct 31 00:37:24.665565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.665472 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" Oct 31 00:37:24.665565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.665510 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" event={"ID":"04a2e696-4d5d-4a8c-86a7-b355b0267363","Type":"ContainerDied","Data":"82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7"} Oct 31 00:37:24.665565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.665547 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6" event={"ID":"04a2e696-4d5d-4a8c-86a7-b355b0267363","Type":"ContainerDied","Data":"bdadb73aa16133ec7b7813b7b47e44c144f711eac24166a81685b4c1361bb6f9"} Oct 31 00:37:24.665565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.665563 5073 scope.go:122] "RemoveContainer" containerID="82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7" Oct 31 00:37:24.674768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.674747 5073 scope.go:122] "RemoveContainer" containerID="82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7" Oct 31 00:37:24.675061 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:24.675041 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7\": container with ID starting with 82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7 not found: ID does not exist" containerID="82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7" Oct 31 00:37:24.675126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.675074 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7"} err="failed to get container status \"82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7\": rpc error: code = NotFound desc = could not find container \"82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7\": container with ID starting with 82e688b9a7b3478266a4039d4fbe54bd9909d4c4cc9541dd46c5ee87986d75b7 not found: ID does not exist" Oct 31 00:37:24.685898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.685860 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.685849001 podStartE2EDuration="1.685849001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:24.683962845 +0000 UTC m=+2328.744737541" watchObservedRunningTime="2026-10-31 00:37:24.685849001 +0000 UTC m=+2328.746623696" Oct 31 00:37:24.700930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.700903 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6"] Oct 31 00:37:24.709358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:24.709335 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-666b69549-8qhp6"] Oct 31 00:37:26.058436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:26.058402 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a2e696-4d5d-4a8c-86a7-b355b0267363" path="/var/lib/kubelet/pods/04a2e696-4d5d-4a8c-86a7-b355b0267363/volumes" Oct 31 00:37:35.289002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.288964 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:35.289433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.289183 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" containerName="otc-container" containerID="cri-o://28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb" gracePeriod=30 Oct 31 00:37:35.538136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.538114 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.634805 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.634722 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert\") pod \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " Oct 31 00:37:35.634942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.634812 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs\") pod \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " Oct 31 00:37:35.634942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.634862 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs\") pod \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " Oct 31 00:37:35.634942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.634895 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal\") pod \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " Oct 31 00:37:35.634942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.634928 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7\" (UniqueName: \"kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7\") pod \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " Oct 31 00:37:35.635108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.634945 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs\") pod \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\" (UID: \"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0\") " Oct 31 00:37:35.635293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.635267 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal" pod "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" (UID: "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:35.637100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.637076 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert" pod "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" (UID: "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:35.637209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.637182 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs" pod "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" (UID: "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:35.637319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.637296 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs" pod "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" (UID: "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:35.637541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.637526 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs" pod "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" (UID: "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:35.637595 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.637550 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7" pod "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" (UID: "c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0"). InnerVolumeSpecName "kube-api-access-phfm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:35.722344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.722313 5073 generic.go:356] "Generic (PLEG): container finished" podID="c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" containerID="28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb" exitCode=0 Oct 31 00:37:35.722516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.722367 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.722516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.722410 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0","Type":"ContainerDied","Data":"28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb"} Oct 31 00:37:35.722516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.722476 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0","Type":"ContainerDied","Data":"970b96b56487ab44e96e4ce9e2e95d3b420115494147d3ed14132e27849b961c"} Oct 31 00:37:35.722516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.722506 5073 scope.go:122] "RemoveContainer" containerID="28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb" Oct 31 00:37:35.731671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.731650 5073 scope.go:122] "RemoveContainer" containerID="28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb" Oct 31 00:37:35.731932 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:35.731916 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb\": container with ID starting with 28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb not found: ID does not exist" containerID="28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb" Oct 31 00:37:35.731975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.731938 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb"} err="failed to get container status \"28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb\": rpc error: code = NotFound desc = could not find container \"28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb\": container with ID starting with 28ca730cc3b304898d9daa74c42a780a31c6ec600e0aa5cb77d8830f75f89bbb not found: ID does not exist" Oct 31 00:37:35.736187 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.736168 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.736262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.736189 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.736262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.736199 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.736262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.736209 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.736262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.736218 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phfm7\" (UniqueName: \"kubernetes.io/projected/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-kube-api-access-phfm7\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.736262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.736226 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:35.743925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.743904 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:35.748243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.748223 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:35.779502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779480 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:35.779855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779841 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" containerName="otc-container" Oct 31 00:37:35.779904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779856 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" containerName="otc-container" Oct 31 00:37:35.779904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779875 5073 cpu_manager.go:414] "containerMap: removing container" podUID="04a2e696-4d5d-4a8c-86a7-b355b0267363" containerName="ta-container" Oct 31 00:37:35.779904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779880 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a2e696-4d5d-4a8c-86a7-b355b0267363" containerName="ta-container" Oct 31 00:37:35.779998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779936 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" containerName="otc-container" Oct 31 00:37:35.779998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.779945 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a2e696-4d5d-4a8c-86a7-b355b0267363" containerName="ta-container" Oct 31 00:37:35.784249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.784235 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.788944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.788923 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:35.789047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.788948 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:35.803945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.803922 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:35.804207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.803973 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:35.804207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.804038 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:35.806107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.806084 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:35.813677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.813659 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:37:35.837441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.837412 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.837441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.837441 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.837619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.837470 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmd9\" (UniqueName: \"kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.837619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.837589 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.837735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.837625 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.837735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.837649 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.938494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.938439 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.938494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.938468 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.938641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.938501 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmd9\" (UniqueName: \"kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.938641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.938538 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.938641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.938564 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.938641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.938586 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.939511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.939482 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.941204 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.941178 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.941308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.941270 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.941591 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.941572 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.941656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.941642 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:35.953915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:35.953895 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmd9\" (UniqueName: \"kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9\") pod \"data-science-collector-collector-1\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:36.058751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.058720 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0" path="/var/lib/kubelet/pods/c07d9dd6-e6a7-4c5e-a466-9a16d3b95eb0/volumes" Oct 31 00:37:36.098778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.098759 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:36.107070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.107052 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:36.273647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.273614 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d"] Oct 31 00:37:36.273863 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.273835 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" podUID="9c41f56e-3509-4e95-8849-02ba05415a1f" containerName="ta-container" containerID="cri-o://5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42" gracePeriod=30 Oct 31 00:37:36.387605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.387580 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:36.390833 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:37:36.390808 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8587e9f9_79e1_4f36_93d9_bd8bd0c52ba5.slice/crio-2635cee1bb7cea0593e1c9ed0db3fea1f64b4337605d848bb3fea2838cb2b747 WatchSource:0}: Error finding container 2635cee1bb7cea0593e1c9ed0db3fea1f64b4337605d848bb3fea2838cb2b747: Status 404 returned error can't find the container with id 2635cee1bb7cea0593e1c9ed0db3fea1f64b4337605d848bb3fea2838cb2b747 Oct 31 00:37:36.393503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.393469 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:36.393564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.393547 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:36.522440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.522419 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:36.544114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.544038 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal\") pod \"9c41f56e-3509-4e95-8849-02ba05415a1f\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " Oct 31 00:37:36.544257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.544119 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert\") pod \"9c41f56e-3509-4e95-8849-02ba05415a1f\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " Oct 31 00:37:36.544257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.544160 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569\" (UniqueName: \"kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569\") pod \"9c41f56e-3509-4e95-8849-02ba05415a1f\" (UID: \"9c41f56e-3509-4e95-8849-02ba05415a1f\") " Oct 31 00:37:36.554023 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.546559 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal" pod "9c41f56e-3509-4e95-8849-02ba05415a1f" (UID: "9c41f56e-3509-4e95-8849-02ba05415a1f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:36.554023 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.547876 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569" pod "9c41f56e-3509-4e95-8849-02ba05415a1f" (UID: "9c41f56e-3509-4e95-8849-02ba05415a1f"). InnerVolumeSpecName "kube-api-access-9g569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:36.554023 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.548932 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert" pod "9c41f56e-3509-4e95-8849-02ba05415a1f" (UID: "9c41f56e-3509-4e95-8849-02ba05415a1f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:36.646166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.646133 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/9c41f56e-3509-4e95-8849-02ba05415a1f-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:36.646166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.646168 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/9c41f56e-3509-4e95-8849-02ba05415a1f-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:36.646355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.646180 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g569\" (UniqueName: \"kubernetes.io/projected/9c41f56e-3509-4e95-8849-02ba05415a1f-kube-api-access-9g569\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:36.729469 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.729423 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5","Type":"ContainerStarted","Data":"f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70"} Oct 31 00:37:36.729649 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.729518 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5","Type":"ContainerStarted","Data":"2635cee1bb7cea0593e1c9ed0db3fea1f64b4337605d848bb3fea2838cb2b747"} Oct 31 00:37:36.730823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.730771 5073 generic.go:356] "Generic (PLEG): container finished" podID="9c41f56e-3509-4e95-8849-02ba05415a1f" containerID="5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42" exitCode=0 Oct 31 00:37:36.730929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.730822 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" event={"ID":"9c41f56e-3509-4e95-8849-02ba05415a1f","Type":"ContainerDied","Data":"5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42"} Oct 31 00:37:36.730929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.730846 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" Oct 31 00:37:36.730929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.730862 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d" event={"ID":"9c41f56e-3509-4e95-8849-02ba05415a1f","Type":"ContainerDied","Data":"62d6c3604eb032f749cfdd7ce2e9359dd3814c18bfd16d4545d8879a12f4d6c5"} Oct 31 00:37:36.730929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.730879 5073 scope.go:122] "RemoveContainer" containerID="5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42" Oct 31 00:37:36.739910 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.739894 5073 scope.go:122] "RemoveContainer" containerID="5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42" Oct 31 00:37:36.740207 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:36.740187 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42\": container with ID starting with 5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42 not found: ID does not exist" containerID="5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42" Oct 31 00:37:36.740274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.740214 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42"} err="failed to get container status \"5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42\": rpc error: code = NotFound desc = could not find container \"5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42\": container with ID starting with 5c8e1a7ff4a23653ec327fd457f996a404b1d8a64350a82a7461223348a8df42 not found: ID does not exist" Oct 31 00:37:36.751455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.751411 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.751396993 podStartE2EDuration="1.751396993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:36.750818941 +0000 UTC m=+2340.811593637" watchObservedRunningTime="2026-10-31 00:37:36.751396993 +0000 UTC m=+2340.812171688" Oct 31 00:37:36.771821 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.771159 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d"] Oct 31 00:37:36.777596 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:36.777567 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5b9cb85c54-xd89d"] Oct 31 00:37:37.070742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:37.070716 5073 scope.go:122] "RemoveContainer" containerID="37631a4b5e5987a521719039c75c64dc7f31ee726dd94b2e86f448185309f855" Oct 31 00:37:38.058608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:38.058569 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c41f56e-3509-4e95-8849-02ba05415a1f" path="/var/lib/kubelet/pods/9c41f56e-3509-4e95-8849-02ba05415a1f/volumes" Oct 31 00:37:43.084767 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:43.084724 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:37:47.434548 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.434511 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:47.434977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.434707 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" containerName="otc-container" containerID="cri-o://f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70" gracePeriod=30 Oct 31 00:37:47.548699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.548662 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc"] Oct 31 00:37:47.549285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.549253 5073 cpu_manager.go:414] "containerMap: removing container" podUID="9c41f56e-3509-4e95-8849-02ba05415a1f" containerName="ta-container" Oct 31 00:37:47.549285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.549275 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c41f56e-3509-4e95-8849-02ba05415a1f" containerName="ta-container" Oct 31 00:37:47.549495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.549356 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c41f56e-3509-4e95-8849-02ba05415a1f" containerName="ta-container" Oct 31 00:37:47.553972 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.553950 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.561697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.561669 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc"] Oct 31 00:37:47.562455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.562432 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-gpb29\"" Oct 31 00:37:47.562568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.562533 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:37:47.569855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.569837 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:37:47.634297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.634266 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.634465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.634392 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wpvg\" (UniqueName: \"kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.634465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.634436 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.735107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.735063 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.735296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.735132 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.735296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.735187 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wpvg\" (UniqueName: \"kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.736058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.736021 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.738661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.738632 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.751106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.751070 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wpvg\" (UniqueName: \"kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg\") pod \"data-science-collector-targetallocator-564b4c8d4d-fskqc\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:47.869114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:47.869077 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:48.145534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.145464 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc"] Oct 31 00:37:48.148929 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:37:48.148897 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod101ebe9b_2dce_48ee_8a20_f9694ec45753.slice/crio-d76678b8ceb985fc2a9b7f163d7065927b6c3dcfba30c9562122e70e49e1ecc4 WatchSource:0}: Error finding container d76678b8ceb985fc2a9b7f163d7065927b6c3dcfba30c9562122e70e49e1ecc4: Status 404 returned error can't find the container with id d76678b8ceb985fc2a9b7f163d7065927b6c3dcfba30c9562122e70e49e1ecc4 Oct 31 00:37:48.150974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.150940 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:48.151081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.151008 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:37:48.792090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.792050 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" event={"ID":"101ebe9b-2dce-48ee-8a20-f9694ec45753","Type":"ContainerStarted","Data":"5ae538092778add8a4f891577320f19639d3da3a1518d97bf760c1bf421d2a77"} Oct 31 00:37:48.792090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.792095 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" event={"ID":"101ebe9b-2dce-48ee-8a20-f9694ec45753","Type":"ContainerStarted","Data":"d76678b8ceb985fc2a9b7f163d7065927b6c3dcfba30c9562122e70e49e1ecc4"} Oct 31 00:37:48.792530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.792234 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:48.793763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.793740 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:48.810990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:48.810943 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" podStartSLOduration=1.8109277700000002 podStartE2EDuration="1.81092777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:48.809965539 +0000 UTC m=+2352.870740235" watchObservedRunningTime="2026-10-31 00:37:48.81092777 +0000 UTC m=+2352.871702466" Oct 31 00:37:49.784232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.784209 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:49.798401 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.798376 5073 generic.go:356] "Generic (PLEG): container finished" podID="8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" containerID="f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70" exitCode=0 Oct 31 00:37:49.798710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.798416 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:49.798710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.798458 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5","Type":"ContainerDied","Data":"f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70"} Oct 31 00:37:49.798710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.798509 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5","Type":"ContainerDied","Data":"2635cee1bb7cea0593e1c9ed0db3fea1f64b4337605d848bb3fea2838cb2b747"} Oct 31 00:37:49.798710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.798533 5073 scope.go:122] "RemoveContainer" containerID="f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70" Oct 31 00:37:49.808221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.808196 5073 scope.go:122] "RemoveContainer" containerID="f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70" Oct 31 00:37:49.808511 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:49.808489 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70\": container with ID starting with f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70 not found: ID does not exist" containerID="f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70" Oct 31 00:37:49.808613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.808521 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70"} err="failed to get container status \"f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70\": rpc error: code = NotFound desc = could not find container \"f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70\": container with ID starting with f88d1df5e858bb8d8c67bfe6ba09991888964b65ea7ec9c6dc64fffa29f6ba70 not found: ID does not exist" Oct 31 00:37:49.851942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.851916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert\") pod \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " Oct 31 00:37:49.852060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.851969 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal\") pod \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " Oct 31 00:37:49.852060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.852031 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs\") pod \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " Oct 31 00:37:49.852159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.852059 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs\") pod \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " Oct 31 00:37:49.852159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.852089 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs\") pod \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " Oct 31 00:37:49.852450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.852287 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9\" (UniqueName: \"kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9\") pod \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\" (UID: \"8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5\") " Oct 31 00:37:49.852450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.852403 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal" pod "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" (UID: "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:49.853115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.852694 5073 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otc-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.854424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.854403 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert" pod "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" (UID: "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.854518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.854423 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs" pod "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" (UID: "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.854518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.854504 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs" pod "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" (UID: "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.854908 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.854886 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs" pod "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" (UID: "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.854993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.854907 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9" pod "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" (UID: "8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5"). InnerVolumeSpecName "kube-api-access-jkmd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:49.954057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.954033 5073 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-otlp-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.954057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.954055 5073 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-monitoring-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.954189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.954064 5073 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-tls-certs\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.954189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.954073 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkmd9\" (UniqueName: \"kubernetes.io/projected/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-kube-api-access-jkmd9\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.954189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:49.954092 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5-data-science-collector-ta-client-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:50.114170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:50.114142 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:50.118509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:50.118488 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:52.058510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:52.058477 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" path="/var/lib/kubelet/pods/8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5/volumes" Oct 31 00:37:53.345956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.345917 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j"] Oct 31 00:37:53.346560 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.346154 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" podUID="7e51c022-4169-4c3e-a781-333812b1dcfb" containerName="thanos-querier" containerID="cri-o://43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a" gracePeriod=30 Oct 31 00:37:53.440017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.439984 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:37:53.440573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.440283 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="prometheus" containerID="cri-o://212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" gracePeriod=600 Oct 31 00:37:53.440573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.440325 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="thanos-sidecar" containerID="cri-o://c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" gracePeriod=600 Oct 31 00:37:53.440573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.440361 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="config-reloader" containerID="cri-o://440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" gracePeriod=600 Oct 31 00:37:53.641839 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:53.641801 5073 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430594d8_210a_4871_80be_e8afcc0ce495.slice/crio-440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:37:53.662948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.662911 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc"] Oct 31 00:37:53.663296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.663267 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" podUID="101ebe9b-2dce-48ee-8a20-f9694ec45753" containerName="ta-container" containerID="cri-o://5ae538092778add8a4f891577320f19639d3da3a1518d97bf760c1bf421d2a77" gracePeriod=30 Oct 31 00:37:53.800675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.800643 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:37:53.814997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.814968 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:37:53.828655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.828623 5073 generic.go:356] "Generic (PLEG): container finished" podID="7e51c022-4169-4c3e-a781-333812b1dcfb" containerID="43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a" exitCode=0 Oct 31 00:37:53.828808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.828710 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" event={"ID":"7e51c022-4169-4c3e-a781-333812b1dcfb","Type":"ContainerDied","Data":"43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a"} Oct 31 00:37:53.828808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.828745 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" event={"ID":"7e51c022-4169-4c3e-a781-333812b1dcfb","Type":"ContainerDied","Data":"83affdfbb29a2642b75c0d4271997c8ed9feaa8a760bff4ef37cf5586eaf2129"} Oct 31 00:37:53.828808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.828767 5073 scope.go:122] "RemoveContainer" containerID="43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a" Oct 31 00:37:53.828966 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.828949 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j" Oct 31 00:37:53.833170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.832743 5073 generic.go:356] "Generic (PLEG): container finished" podID="101ebe9b-2dce-48ee-8a20-f9694ec45753" containerID="5ae538092778add8a4f891577320f19639d3da3a1518d97bf760c1bf421d2a77" exitCode=0 Oct 31 00:37:53.833170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.832883 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" event={"ID":"101ebe9b-2dce-48ee-8a20-f9694ec45753","Type":"ContainerDied","Data":"5ae538092778add8a4f891577320f19639d3da3a1518d97bf760c1bf421d2a77"} Oct 31 00:37:53.836741 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.836704 5073 generic.go:356] "Generic (PLEG): container finished" podID="430594d8-210a-4871-80be-e8afcc0ce495" containerID="c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" exitCode=0 Oct 31 00:37:53.836914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.836894 5073 generic.go:356] "Generic (PLEG): container finished" podID="430594d8-210a-4871-80be-e8afcc0ce495" containerID="440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" exitCode=0 Oct 31 00:37:53.837032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.837019 5073 generic.go:356] "Generic (PLEG): container finished" podID="430594d8-210a-4871-80be-e8afcc0ce495" containerID="212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" exitCode=0 Oct 31 00:37:53.837315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.837285 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerDied","Data":"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6"} Oct 31 00:37:53.837527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.837508 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerDied","Data":"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b"} Oct 31 00:37:53.837726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.837694 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerDied","Data":"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6"} Oct 31 00:37:53.837910 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.837884 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"430594d8-210a-4871-80be-e8afcc0ce495","Type":"ContainerDied","Data":"1bb3c80b78616f4e1ab008468a95947db7fa45c026e9154da497f51e3770063b"} Oct 31 00:37:53.838183 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.838168 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:37:53.848665 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.848581 5073 scope.go:122] "RemoveContainer" containerID="43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a" Oct 31 00:37:53.849036 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:53.849013 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a\": container with ID starting with 43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a not found: ID does not exist" containerID="43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a" Oct 31 00:37:53.849134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.849044 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a"} err="failed to get container status \"43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a\": rpc error: code = NotFound desc = could not find container \"43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a\": container with ID starting with 43b39957222ff3f022970e585566fab9f70e5c2937d911447daa8453d823e53a not found: ID does not exist" Oct 31 00:37:53.849134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.849062 5073 scope.go:122] "RemoveContainer" containerID="c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" Oct 31 00:37:53.862759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.862650 5073 scope.go:122] "RemoveContainer" containerID="440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" Oct 31 00:37:53.874344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.874319 5073 scope.go:122] "RemoveContainer" containerID="212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" Oct 31 00:37:53.886741 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886712 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.886883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886766 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.886883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886832 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.886883 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886869 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886908 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm\" (UniqueName: \"kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm\") pod \"7e51c022-4169-4c3e-a781-333812b1dcfb\" (UID: \"7e51c022-4169-4c3e-a781-333812b1dcfb\") " Oct 31 00:37:53.887066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886943 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.886984 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887009 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887041 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887101 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out\" (UniqueName: \"kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887137 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887168 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887263 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.887354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.887308 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6\") pod \"430594d8-210a-4871-80be-e8afcc0ce495\" (UID: \"430594d8-210a-4871-80be-e8afcc0ce495\") " Oct 31 00:37:53.888987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.888964 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.889192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.889165 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.889316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.889300 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.889416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.889394 5073 scope.go:122] "RemoveContainer" containerID="93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e" Oct 31 00:37:53.893463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.893430 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.894406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.894333 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.894406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.894368 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.894406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.894379 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.894551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.894431 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm" pod "7e51c022-4169-4c3e-a781-333812b1dcfb" (UID: "7e51c022-4169-4c3e-a781-333812b1dcfb"). InnerVolumeSpecName "kube-api-access-kzrpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.894842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.894821 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.895091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.895071 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:37:53.895332 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.895313 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.895563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.895544 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "kube-api-access-gpdt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.901928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.900414 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:37:53.902881 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.902861 5073 scope.go:122] "RemoveContainer" containerID="c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" Oct 31 00:37:53.903201 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:53.903168 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": container with ID starting with c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6 not found: ID does not exist" containerID="c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" Oct 31 00:37:53.903306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.903208 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6"} err="failed to get container status \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": rpc error: code = NotFound desc = could not find container \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": container with ID starting with c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6 not found: ID does not exist" Oct 31 00:37:53.903306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.903232 5073 scope.go:122] "RemoveContainer" containerID="440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" Oct 31 00:37:53.903547 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:53.903522 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": container with ID starting with 440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b not found: ID does not exist" containerID="440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" Oct 31 00:37:53.903597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.903555 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b"} err="failed to get container status \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": rpc error: code = NotFound desc = could not find container \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": container with ID starting with 440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b not found: ID does not exist" Oct 31 00:37:53.903597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.903575 5073 scope.go:122] "RemoveContainer" containerID="212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" Oct 31 00:37:53.904019 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:53.903997 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": container with ID starting with 212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6 not found: ID does not exist" containerID="212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" Oct 31 00:37:53.904114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904025 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6"} err="failed to get container status \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": rpc error: code = NotFound desc = could not find container \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": container with ID starting with 212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6 not found: ID does not exist" Oct 31 00:37:53.904114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904043 5073 scope.go:122] "RemoveContainer" containerID="93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e" Oct 31 00:37:53.904359 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:37:53.904329 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": container with ID starting with 93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e not found: ID does not exist" containerID="93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e" Oct 31 00:37:53.904437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904354 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e"} err="failed to get container status \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": rpc error: code = NotFound desc = could not find container \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": container with ID starting with 93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e not found: ID does not exist" Oct 31 00:37:53.904437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904376 5073 scope.go:122] "RemoveContainer" containerID="c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" Oct 31 00:37:53.904683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904661 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6"} err="failed to get container status \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": rpc error: code = NotFound desc = could not find container \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": container with ID starting with c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6 not found: ID does not exist" Oct 31 00:37:53.904759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904684 5073 scope.go:122] "RemoveContainer" containerID="440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" Oct 31 00:37:53.904963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904940 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b"} err="failed to get container status \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": rpc error: code = NotFound desc = could not find container \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": container with ID starting with 440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b not found: ID does not exist" Oct 31 00:37:53.905054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.904964 5073 scope.go:122] "RemoveContainer" containerID="212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" Oct 31 00:37:53.905242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.905221 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6"} err="failed to get container status \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": rpc error: code = NotFound desc = could not find container \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": container with ID starting with 212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6 not found: ID does not exist" Oct 31 00:37:53.905328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.905243 5073 scope.go:122] "RemoveContainer" containerID="93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e" Oct 31 00:37:53.905549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.905530 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e"} err="failed to get container status \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": rpc error: code = NotFound desc = could not find container \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": container with ID starting with 93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e not found: ID does not exist" Oct 31 00:37:53.905628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.905551 5073 scope.go:122] "RemoveContainer" containerID="c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6" Oct 31 00:37:53.905874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.905852 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6"} err="failed to get container status \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": rpc error: code = NotFound desc = could not find container \"c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6\": container with ID starting with c07e99cdf7f08c7d772e9ebdd299f4bd99a7a2a828885b882e5affea333eaac6 not found: ID does not exist" Oct 31 00:37:53.905962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.905875 5073 scope.go:122] "RemoveContainer" containerID="440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b" Oct 31 00:37:53.906344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.906239 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b"} err="failed to get container status \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": rpc error: code = NotFound desc = could not find container \"440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b\": container with ID starting with 440a1e51fbfb9039924b67fd83a1bdfe60094b06157660d81d7bf8f0c4d3fe5b not found: ID does not exist" Oct 31 00:37:53.906344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.906274 5073 scope.go:122] "RemoveContainer" containerID="212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6" Oct 31 00:37:53.906856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.906717 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6"} err="failed to get container status \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": rpc error: code = NotFound desc = could not find container \"212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6\": container with ID starting with 212f043b14c7ebf091e4e13189ff6b45b9791e9f10c2bed075e198231cac43e6 not found: ID does not exist" Oct 31 00:37:53.906856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.906740 5073 scope.go:122] "RemoveContainer" containerID="93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e" Oct 31 00:37:53.907303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.907228 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e"} err="failed to get container status \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": rpc error: code = NotFound desc = could not find container \"93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e\": container with ID starting with 93063f317e1d2fca4c11b53b48f6c04794bb188137bc121f9df501db28475c1e not found: ID does not exist" Oct 31 00:37:53.916162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.916091 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config" pod "430594d8-210a-4871-80be-e8afcc0ce495" (UID: "430594d8-210a-4871-80be-e8afcc0ce495"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.988493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988462 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430594d8-210a-4871-80be-e8afcc0ce495-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988497 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988519 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988560 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:37:53.988618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988577 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdt6\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-kube-api-access-gpdt6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988591 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988618 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988607 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988623 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988640 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988655 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrpm\" (UniqueName: \"kubernetes.io/projected/7e51c022-4169-4c3e-a781-333812b1dcfb-kube-api-access-kzrpm\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988669 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988684 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430594d8-210a-4871-80be-e8afcc0ce495-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988697 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/430594d8-210a-4871-80be-e8afcc0ce495-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.988909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.988712 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/430594d8-210a-4871-80be-e8afcc0ce495-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.990805 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.990770 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:53.999713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:53.999696 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:37:54.089592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.089571 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert\") pod \"101ebe9b-2dce-48ee-8a20-f9694ec45753\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " Oct 31 00:37:54.089699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.089609 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal\" (UniqueName: \"kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal\") pod \"101ebe9b-2dce-48ee-8a20-f9694ec45753\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " Oct 31 00:37:54.089699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.089646 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg\" (UniqueName: \"kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg\") pod \"101ebe9b-2dce-48ee-8a20-f9694ec45753\" (UID: \"101ebe9b-2dce-48ee-8a20-f9694ec45753\") " Oct 31 00:37:54.089859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.089844 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" Oct 31 00:37:54.089938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.089901 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal" pod "101ebe9b-2dce-48ee-8a20-f9694ec45753" (UID: "101ebe9b-2dce-48ee-8a20-f9694ec45753"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:54.091697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.091672 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert" pod "101ebe9b-2dce-48ee-8a20-f9694ec45753" (UID: "101ebe9b-2dce-48ee-8a20-f9694ec45753"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.091847 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.091745 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg" pod "101ebe9b-2dce-48ee-8a20-f9694ec45753" (UID: "101ebe9b-2dce-48ee-8a20-f9694ec45753"). InnerVolumeSpecName "kube-api-access-8wpvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.144242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.144195 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j"] Oct 31 00:37:54.150359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.150340 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-fgh7j"] Oct 31 00:37:54.163648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.163629 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:37:54.168702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.168679 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:37:54.190513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.190487 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/101ebe9b-2dce-48ee-8a20-f9694ec45753-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.190589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.190514 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/101ebe9b-2dce-48ee-8a20-f9694ec45753-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.190589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.190530 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wpvg\" (UniqueName: \"kubernetes.io/projected/101ebe9b-2dce-48ee-8a20-f9694ec45753-kube-api-access-8wpvg\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.845063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.845031 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" Oct 31 00:37:54.845493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.845028 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc" event={"ID":"101ebe9b-2dce-48ee-8a20-f9694ec45753","Type":"ContainerDied","Data":"d76678b8ceb985fc2a9b7f163d7065927b6c3dcfba30c9562122e70e49e1ecc4"} Oct 31 00:37:54.845493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.845171 5073 scope.go:122] "RemoveContainer" containerID="5ae538092778add8a4f891577320f19639d3da3a1518d97bf760c1bf421d2a77" Oct 31 00:37:54.867534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.867510 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc"] Oct 31 00:37:54.871273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:54.871251 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-564b4c8d4d-fskqc"] Oct 31 00:37:56.057795 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:56.057761 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101ebe9b-2dce-48ee-8a20-f9694ec45753" path="/var/lib/kubelet/pods/101ebe9b-2dce-48ee-8a20-f9694ec45753/volumes" Oct 31 00:37:56.058213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:56.058199 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430594d8-210a-4871-80be-e8afcc0ce495" path="/var/lib/kubelet/pods/430594d8-210a-4871-80be-e8afcc0ce495/volumes" Oct 31 00:37:56.058614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:37:56.058602 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e51c022-4169-4c3e-a781-333812b1dcfb" path="/var/lib/kubelet/pods/7e51c022-4169-4c3e-a781-333812b1dcfb/volumes" Oct 31 00:38:01.250468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250439 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd"] Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250807 5073 cpu_manager.go:414] "containerMap: removing container" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="init-config-reloader" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250820 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="init-config-reloader" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250838 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7e51c022-4169-4c3e-a781-333812b1dcfb" containerName="thanos-querier" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250843 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e51c022-4169-4c3e-a781-333812b1dcfb" containerName="thanos-querier" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250854 5073 cpu_manager.go:414] "containerMap: removing container" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="config-reloader" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250859 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="config-reloader" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250866 5073 cpu_manager.go:414] "containerMap: removing container" podUID="8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" containerName="otc-container" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250873 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" containerName="otc-container" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250885 5073 cpu_manager.go:414] "containerMap: removing container" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="thanos-sidecar" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250890 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="thanos-sidecar" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250898 5073 cpu_manager.go:414] "containerMap: removing container" podUID="101ebe9b-2dce-48ee-8a20-f9694ec45753" containerName="ta-container" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250902 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101ebe9b-2dce-48ee-8a20-f9694ec45753" containerName="ta-container" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250915 5073 cpu_manager.go:414] "containerMap: removing container" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="prometheus" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250919 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="prometheus" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250977 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="config-reloader" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250985 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="8587e9f9-79e1-4f36-93d9-bd8bd0c52ba5" containerName="otc-container" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250991 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="thanos-sidecar" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.250996 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="101ebe9b-2dce-48ee-8a20-f9694ec45753" containerName="ta-container" Oct 31 00:38:01.250993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.251002 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="430594d8-210a-4871-80be-e8afcc0ce495" containerName="prometheus" Oct 31 00:38:01.252290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.251010 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e51c022-4169-4c3e-a781-333812b1dcfb" containerName="thanos-querier" Oct 31 00:38:01.253941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.253922 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:01.261958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.261933 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:38:01.262082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.262041 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:38:01.262149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.262135 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:38:01.262496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.262470 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd"] Oct 31 00:38:01.339547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.339522 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4pb\" (UniqueName: \"kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd\" (UID: \"c77a46d6-a227-4023-be3c-6e11c0cc3d51\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:01.439916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.439891 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4pb\" (UniqueName: \"kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd\" (UID: \"c77a46d6-a227-4023-be3c-6e11c0cc3d51\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:01.452518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.452493 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4pb\" (UniqueName: \"kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd\" (UID: \"c77a46d6-a227-4023-be3c-6e11c0cc3d51\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:01.567635 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.567544 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:01.636594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.636560 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js"] Oct 31 00:38:01.642265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.641699 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.647833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.646099 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:38:01.657511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.653828 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:38:01.657511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.655151 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js"] Oct 31 00:38:01.744711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.744682 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.744876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.744719 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdh7\" (UniqueName: \"kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.744876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.744752 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.745965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.745947 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-2287h\"" Oct 31 00:38:01.845435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.845364 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.845561 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.845432 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdh7\" (UniqueName: \"kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.845865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.845841 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.846550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.846526 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.847296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.847274 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd"] Oct 31 00:38:01.849131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.849106 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.850578 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:38:01.850545 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc77a46d6_a227_4023_be3c_6e11c0cc3d51.slice/crio-34020dc1c5634a241c536cafbc1f7029881a2d3d8f6f798d514d228a9e40833f WatchSource:0}: Error finding container 34020dc1c5634a241c536cafbc1f7029881a2d3d8f6f798d514d228a9e40833f: Status 404 returned error can't find the container with id 34020dc1c5634a241c536cafbc1f7029881a2d3d8f6f798d514d228a9e40833f Oct 31 00:38:01.858455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.858433 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdh7\" (UniqueName: \"kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-956js\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:01.881282 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.881260 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" event={"ID":"c77a46d6-a227-4023-be3c-6e11c0cc3d51","Type":"ContainerStarted","Data":"34020dc1c5634a241c536cafbc1f7029881a2d3d8f6f798d514d228a9e40833f"} Oct 31 00:38:01.966327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:01.966296 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:02.242098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.242068 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js"] Oct 31 00:38:02.245129 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:38:02.245104 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a4be4d_78c9_432f_bea1_385bc8a5a592.slice/crio-1f2bdcf88ab9f7737f50e6050c20a38a07b2e435f70ff5d22f33cda2087367ff WatchSource:0}: Error finding container 1f2bdcf88ab9f7737f50e6050c20a38a07b2e435f70ff5d22f33cda2087367ff: Status 404 returned error can't find the container with id 1f2bdcf88ab9f7737f50e6050c20a38a07b2e435f70ff5d22f33cda2087367ff Oct 31 00:38:02.247236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.247202 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:38:02.247360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.247257 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:38:02.887779 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.887744 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" event={"ID":"c77a46d6-a227-4023-be3c-6e11c0cc3d51","Type":"ContainerStarted","Data":"6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405"} Oct 31 00:38:02.889317 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.889291 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" event={"ID":"71a4be4d-78c9-432f-bea1-385bc8a5a592","Type":"ContainerStarted","Data":"772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b"} Oct 31 00:38:02.889435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.889324 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" event={"ID":"71a4be4d-78c9-432f-bea1-385bc8a5a592","Type":"ContainerStarted","Data":"1f2bdcf88ab9f7737f50e6050c20a38a07b2e435f70ff5d22f33cda2087367ff"} Oct 31 00:38:02.889502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.889478 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:02.891827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.891810 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:02.905764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.905716 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" podStartSLOduration=1.905705705 podStartE2EDuration="1.905705705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:02.904334622 +0000 UTC m=+2366.965109318" watchObservedRunningTime="2026-10-31 00:38:02.905705705 +0000 UTC m=+2366.966480394" Oct 31 00:38:02.925795 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:02.925730 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" podStartSLOduration=1.9257122340000001 podStartE2EDuration="1.925712234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:02.923828797 +0000 UTC m=+2366.984603492" watchObservedRunningTime="2026-10-31 00:38:02.925712234 +0000 UTC m=+2366.986486929" Oct 31 00:38:06.911669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:06.911623 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk"] Oct 31 00:38:06.915098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:06.915081 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:38:06.922909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:06.922757 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk"] Oct 31 00:38:06.987405 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:06.987368 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz5jv\" (UniqueName: \"kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk\" (UID: \"d59eecd3-7750-4b71-ac82-dc9c85d74ea7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:38:07.087904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.087871 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz5jv\" (UniqueName: \"kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk\" (UID: \"d59eecd3-7750-4b71-ac82-dc9c85d74ea7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:38:07.101232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.101205 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz5jv\" (UniqueName: \"kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk\" (UID: \"d59eecd3-7750-4b71-ac82-dc9c85d74ea7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:38:07.229042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.228958 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:38:07.506226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.506194 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk"] Oct 31 00:38:07.510451 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:38:07.510419 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd59eecd3_7750_4b71_ac82_dc9c85d74ea7.slice/crio-360aff1236d76779fbc4374581de089994086005cf81fa807cc3eefa8cd2c72c WatchSource:0}: Error finding container 360aff1236d76779fbc4374581de089994086005cf81fa807cc3eefa8cd2c72c: Status 404 returned error can't find the container with id 360aff1236d76779fbc4374581de089994086005cf81fa807cc3eefa8cd2c72c Oct 31 00:38:07.850372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.850293 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:38:07.854049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.854031 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.869809 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.869769 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:38:07.914715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.914686 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" event={"ID":"d59eecd3-7750-4b71-ac82-dc9c85d74ea7","Type":"ContainerStarted","Data":"52155080d15f62e16239053ce07c663e368385353be893d219e0321b7fc2eb8e"} Oct 31 00:38:07.915120 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.914728 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" event={"ID":"d59eecd3-7750-4b71-ac82-dc9c85d74ea7","Type":"ContainerStarted","Data":"360aff1236d76779fbc4374581de089994086005cf81fa807cc3eefa8cd2c72c"} Oct 31 00:38:07.932554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.932509 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" podStartSLOduration=1.932497356 podStartE2EDuration="1.932497356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:07.932363515 +0000 UTC m=+2371.993138206" watchObservedRunningTime="2026-10-31 00:38:07.932497356 +0000 UTC m=+2371.993272066" Oct 31 00:38:07.934889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934859 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:38:07.934889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934882 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:38:07.935043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934899 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-r54vf\"" Oct 31 00:38:07.935043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934901 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:38:07.935043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934920 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:38:07.935043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934928 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:38:07.935043 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.934920 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:38:07.936619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.936605 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:38:07.936670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.936632 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:38:07.936713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.936693 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:38:07.958677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.958655 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd"] Oct 31 00:38:07.958882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.958837 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" podUID="c77a46d6-a227-4023-be3c-6e11c0cc3d51" containerName="thanos-querier" containerID="cri-o://6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405" gracePeriod=30 Oct 31 00:38:07.997108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997081 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997123 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997147 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997268 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997331 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997354 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997442 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997409 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997474 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997441 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997512 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997469 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997512 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997590 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997572 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:07.997629 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:07.997596 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbk9g\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.098833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.098774 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.098979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.098853 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.098979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.098884 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.098979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.098915 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099041 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099085 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099141 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099118 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099150 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbk9g\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099253 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099297 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099332 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099394 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099441 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.099857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.099831 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.100053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.100024 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.102970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.102501 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.102970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.102585 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.102970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.102660 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.102970 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.102898 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.103686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.103644 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.104278 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.104233 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.104445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.104423 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.104735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.104717 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.112550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.112523 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbk9g\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:08.191453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.191424 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:08.300710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.300678 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb\" (UniqueName: \"kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb\") pod \"c77a46d6-a227-4023-be3c-6e11c0cc3d51\" (UID: \"c77a46d6-a227-4023-be3c-6e11c0cc3d51\") " Oct 31 00:38:08.302900 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.302879 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb" pod "c77a46d6-a227-4023-be3c-6e11c0cc3d51" (UID: "c77a46d6-a227-4023-be3c-6e11c0cc3d51"). InnerVolumeSpecName "kube-api-access-bd4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:08.402254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.402195 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4pb\" (UniqueName: \"kubernetes.io/projected/c77a46d6-a227-4023-be3c-6e11c0cc3d51-kube-api-access-bd4pb\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:08.920695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.920664 5073 generic.go:356] "Generic (PLEG): container finished" podID="c77a46d6-a227-4023-be3c-6e11c0cc3d51" containerID="6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405" exitCode=0 Oct 31 00:38:08.921150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.920710 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" Oct 31 00:38:08.921150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.920752 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" event={"ID":"c77a46d6-a227-4023-be3c-6e11c0cc3d51","Type":"ContainerDied","Data":"6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405"} Oct 31 00:38:08.921150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.920820 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd" event={"ID":"c77a46d6-a227-4023-be3c-6e11c0cc3d51","Type":"ContainerDied","Data":"34020dc1c5634a241c536cafbc1f7029881a2d3d8f6f798d514d228a9e40833f"} Oct 31 00:38:08.921150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.920841 5073 scope.go:122] "RemoveContainer" containerID="6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405" Oct 31 00:38:08.929850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.929827 5073 scope.go:122] "RemoveContainer" containerID="6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405" Oct 31 00:38:08.930119 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:38:08.930101 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405\": container with ID starting with 6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405 not found: ID does not exist" containerID="6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405" Oct 31 00:38:08.930190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.930126 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405"} err="failed to get container status \"6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405\": rpc error: code = NotFound desc = could not find container \"6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405\": container with ID starting with 6a920258b194e7c0e71b3b965dbe28d88389ea53d48bb81d8e4a39d7c0198405 not found: ID does not exist" Oct 31 00:38:08.941927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.941903 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd"] Oct 31 00:38:08.946470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:08.946446 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-cqgnd"] Oct 31 00:38:10.059081 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:10.059037 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77a46d6-a227-4023-be3c-6e11c0cc3d51" path="/var/lib/kubelet/pods/c77a46d6-a227-4023-be3c-6e11c0cc3d51/volumes" Oct 31 00:38:13.085585 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:38:13.085543 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:38:14.554069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.554034 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.560803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.560755 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.655216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.655191 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.655347 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.655328 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.658658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.658641 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.749951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.749925 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.765363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.765343 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:14.774700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:14.774682 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:15.062854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:15.062800 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:38:15.067508 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:38:15.067471 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2312322_6a7e_41b4_bf94_c989d27846b2.slice/crio-c767db971282a878ada5a11a244b25eba17ed87972fd352a370dccc05e1c2366 WatchSource:0}: Error finding container c767db971282a878ada5a11a244b25eba17ed87972fd352a370dccc05e1c2366: Status 404 returned error can't find the container with id c767db971282a878ada5a11a244b25eba17ed87972fd352a370dccc05e1c2366 Oct 31 00:38:15.953669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:15.953587 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerStarted","Data":"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2"} Oct 31 00:38:15.953669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:15.953654 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerStarted","Data":"c767db971282a878ada5a11a244b25eba17ed87972fd352a370dccc05e1c2366"} Oct 31 00:38:18.968959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:18.968927 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerID="e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2" exitCode=0 Oct 31 00:38:18.969267 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:18.968990 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerDied","Data":"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2"} Oct 31 00:38:19.976095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:19.976059 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerStarted","Data":"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e"} Oct 31 00:38:19.976438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:19.976124 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerStarted","Data":"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26"} Oct 31 00:38:19.976438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:19.976148 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerStarted","Data":"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a"} Oct 31 00:38:20.010617 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:20.010571 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=9.62477167 podStartE2EDuration="13.010559128s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385787458s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:20.00828947 +0000 UTC m=+2384.069064201" watchObservedRunningTime="2026-10-31 00:38:20.010559128 +0000 UTC m=+2384.071333822" Oct 31 00:38:24.774950 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:24.774920 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:27.833588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.833552 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh"] Oct 31 00:38:27.834014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.833895 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c77a46d6-a227-4023-be3c-6e11c0cc3d51" containerName="thanos-querier" Oct 31 00:38:27.834014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.833907 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77a46d6-a227-4023-be3c-6e11c0cc3d51" containerName="thanos-querier" Oct 31 00:38:27.834014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.833986 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77a46d6-a227-4023-be3c-6e11c0cc3d51" containerName="thanos-querier" Oct 31 00:38:27.837137 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.837121 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:27.846912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.846889 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh"] Oct 31 00:38:27.966054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.966026 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:27.966192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.966091 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:27.966192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:27.966153 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6jr\" (UniqueName: \"kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.067441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.067410 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.067577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.067448 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6jr\" (UniqueName: \"kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.067577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.067483 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.068030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.068008 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.070099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.070078 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.079891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.079862 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6jr\" (UniqueName: \"kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr\") pod \"data-science-collector-targetallocator-5fd7b8db97-gk8gh\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.149690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.149641 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:28.425050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.424975 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh"] Oct 31 00:38:28.427924 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:38:28.427898 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3fe98f1_5bfe_461e_bd81_e47643e7fd2d.slice/crio-3c2a0a887a52b52fe3b7c1e33c6c888abe05f032a07d3d7ec5b2c1a2faf69b30 WatchSource:0}: Error finding container 3c2a0a887a52b52fe3b7c1e33c6c888abe05f032a07d3d7ec5b2c1a2faf69b30: Status 404 returned error can't find the container with id 3c2a0a887a52b52fe3b7c1e33c6c888abe05f032a07d3d7ec5b2c1a2faf69b30 Oct 31 00:38:28.430087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.430058 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:38:28.430160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:28.430112 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:38:29.026550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.026515 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" event={"ID":"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d","Type":"ContainerStarted","Data":"e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09"} Oct 31 00:38:29.026550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.026555 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" event={"ID":"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d","Type":"ContainerStarted","Data":"3c2a0a887a52b52fe3b7c1e33c6c888abe05f032a07d3d7ec5b2c1a2faf69b30"} Oct 31 00:38:29.026992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.026702 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:29.028341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.028319 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:29.046014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.045974 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" podStartSLOduration=2.04596267 podStartE2EDuration="2.04596267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:29.044334564 +0000 UTC m=+2393.105109255" watchObservedRunningTime="2026-10-31 00:38:29.04596267 +0000 UTC m=+2393.106737428" Oct 31 00:38:29.090622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.090594 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js"] Oct 31 00:38:29.090835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.090808 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" podUID="71a4be4d-78c9-432f-bea1-385bc8a5a592" containerName="ta-container" containerID="cri-o://772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b" gracePeriod=30 Oct 31 00:38:29.332931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.332905 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:29.478028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.477991 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal\" (UniqueName: \"kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal\") pod \"71a4be4d-78c9-432f-bea1-385bc8a5a592\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " Oct 31 00:38:29.478175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.478067 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7\" (UniqueName: \"kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7\") pod \"71a4be4d-78c9-432f-bea1-385bc8a5a592\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " Oct 31 00:38:29.478221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.478175 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert\") pod \"71a4be4d-78c9-432f-bea1-385bc8a5a592\" (UID: \"71a4be4d-78c9-432f-bea1-385bc8a5a592\") " Oct 31 00:38:29.478417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.478387 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal" pod "71a4be4d-78c9-432f-bea1-385bc8a5a592" (UID: "71a4be4d-78c9-432f-bea1-385bc8a5a592"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:29.480291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.480270 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7" pod "71a4be4d-78c9-432f-bea1-385bc8a5a592" (UID: "71a4be4d-78c9-432f-bea1-385bc8a5a592"). InnerVolumeSpecName "kube-api-access-5cdh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:29.480375 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.480294 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert" pod "71a4be4d-78c9-432f-bea1-385bc8a5a592" (UID: "71a4be4d-78c9-432f-bea1-385bc8a5a592"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:29.579588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.579537 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cdh7\" (UniqueName: \"kubernetes.io/projected/71a4be4d-78c9-432f-bea1-385bc8a5a592-kube-api-access-5cdh7\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:29.579588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.579559 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/71a4be4d-78c9-432f-bea1-385bc8a5a592-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:29.579588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.579570 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/71a4be4d-78c9-432f-bea1-385bc8a5a592-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:29.774914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.774880 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:29.779701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:29.779682 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:30.033130 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.033101 5073 generic.go:356] "Generic (PLEG): container finished" podID="71a4be4d-78c9-432f-bea1-385bc8a5a592" containerID="772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b" exitCode=0 Oct 31 00:38:30.033563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.033145 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" Oct 31 00:38:30.033563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.033195 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" event={"ID":"71a4be4d-78c9-432f-bea1-385bc8a5a592","Type":"ContainerDied","Data":"772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b"} Oct 31 00:38:30.033563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.033240 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js" event={"ID":"71a4be4d-78c9-432f-bea1-385bc8a5a592","Type":"ContainerDied","Data":"1f2bdcf88ab9f7737f50e6050c20a38a07b2e435f70ff5d22f33cda2087367ff"} Oct 31 00:38:30.033563 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.033271 5073 scope.go:122] "RemoveContainer" containerID="772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b" Oct 31 00:38:30.038624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.038606 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:38:30.043571 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.043552 5073 scope.go:122] "RemoveContainer" containerID="772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b" Oct 31 00:38:30.043920 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:38:30.043903 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b\": container with ID starting with 772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b not found: ID does not exist" containerID="772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b" Oct 31 00:38:30.043920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.043930 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b"} err="failed to get container status \"772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b\": rpc error: code = NotFound desc = could not find container \"772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b\": container with ID starting with 772ef345f3d9b2dda2133d6589ba4bddff4eab350817cb556390f0152bcd269b not found: ID does not exist" Oct 31 00:38:30.084335 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.084313 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js"] Oct 31 00:38:30.094049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:30.094025 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-956js"] Oct 31 00:38:32.057968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:32.057928 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a4be4d-78c9-432f-bea1-385bc8a5a592" path="/var/lib/kubelet/pods/71a4be4d-78c9-432f-bea1-385bc8a5a592/volumes" Oct 31 00:38:36.312294 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:36.312191 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:38:36.322624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:36.322603 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:38:36.323575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:36.323554 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:38:36.330835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:36.330816 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:38:43.091462 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:38:43.091419 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:38:45.520061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.520020 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw"] Oct 31 00:38:45.520580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.520560 5073 cpu_manager.go:414] "containerMap: removing container" podUID="71a4be4d-78c9-432f-bea1-385bc8a5a592" containerName="ta-container" Oct 31 00:38:45.520655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.520583 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a4be4d-78c9-432f-bea1-385bc8a5a592" containerName="ta-container" Oct 31 00:38:45.520728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.520714 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a4be4d-78c9-432f-bea1-385bc8a5a592" containerName="ta-container" Oct 31 00:38:45.525229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.525207 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.533599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.533514 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw"] Oct 31 00:38:45.611025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.611000 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxb4h\" (UniqueName: \"kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.611148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.611066 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.611148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.611106 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.712305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.712278 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.712425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.712310 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.712425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.712351 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxb4h\" (UniqueName: \"kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.713145 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.713117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.714988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.714965 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.724689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.724666 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxb4h\" (UniqueName: \"kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h\") pod \"data-science-collector-targetallocator-6ffc7c9b4f-frmvw\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:45.837813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:45.837725 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:46.113290 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:46.113178 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw"] Oct 31 00:38:46.116459 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:38:46.116420 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08e54ff8_faf4_4b68_a86c_fb14cde4cabb.slice/crio-d498c80fda7823efdb055061565d4f61bfab82c6cb102a75b75ba522d87a7213 WatchSource:0}: Error finding container d498c80fda7823efdb055061565d4f61bfab82c6cb102a75b75ba522d87a7213: Status 404 returned error can't find the container with id d498c80fda7823efdb055061565d4f61bfab82c6cb102a75b75ba522d87a7213 Oct 31 00:38:46.119038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:46.119007 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:38:46.119142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:46.119063 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:38:47.110938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.110902 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" event={"ID":"08e54ff8-faf4-4b68-a86c-fb14cde4cabb","Type":"ContainerStarted","Data":"ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c"} Oct 31 00:38:47.111341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.110950 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" event={"ID":"08e54ff8-faf4-4b68-a86c-fb14cde4cabb","Type":"ContainerStarted","Data":"d498c80fda7823efdb055061565d4f61bfab82c6cb102a75b75ba522d87a7213"} Oct 31 00:38:47.111341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.110971 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:47.112882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.112857 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:38:47.129340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.129295 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" podStartSLOduration=2.12928351 podStartE2EDuration="2.12928351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:47.128824658 +0000 UTC m=+2411.189599353" watchObservedRunningTime="2026-10-31 00:38:47.12928351 +0000 UTC m=+2411.190058205" Oct 31 00:38:47.176409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.176379 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh"] Oct 31 00:38:47.176574 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.176550 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" podUID="c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" containerName="ta-container" containerID="cri-o://e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09" gracePeriod=30 Oct 31 00:38:47.419952 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.419930 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:47.528763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.528733 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert\") pod \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " Oct 31 00:38:47.528975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.528772 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr\" (UniqueName: \"kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr\") pod \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " Oct 31 00:38:47.528975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.528814 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal\" (UniqueName: \"kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal\") pod \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\" (UID: \"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d\") " Oct 31 00:38:47.529240 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.529218 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal" pod "c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" (UID: "c3fe98f1-5bfe-461e-bd81-e47643e7fd2d"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:38:47.531127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.531098 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert" pod "c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" (UID: "c3fe98f1-5bfe-461e-bd81-e47643e7fd2d"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:47.531207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.531101 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr" pod "c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" (UID: "c3fe98f1-5bfe-461e-bd81-e47643e7fd2d"). InnerVolumeSpecName "kube-api-access-zv6jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:47.629873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.629847 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:47.629873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.629871 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6jr\" (UniqueName: \"kubernetes.io/projected/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-kube-api-access-zv6jr\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:47.629873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:47.629882 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:48.117708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.117679 5073 generic.go:356] "Generic (PLEG): container finished" podID="c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" containerID="e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09" exitCode=0 Oct 31 00:38:48.118122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.117725 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" Oct 31 00:38:48.118122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.117769 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" event={"ID":"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d","Type":"ContainerDied","Data":"e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09"} Oct 31 00:38:48.118122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.117842 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh" event={"ID":"c3fe98f1-5bfe-461e-bd81-e47643e7fd2d","Type":"ContainerDied","Data":"3c2a0a887a52b52fe3b7c1e33c6c888abe05f032a07d3d7ec5b2c1a2faf69b30"} Oct 31 00:38:48.118122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.117867 5073 scope.go:122] "RemoveContainer" containerID="e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09" Oct 31 00:38:48.127255 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.127240 5073 scope.go:122] "RemoveContainer" containerID="e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09" Oct 31 00:38:48.127519 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:38:48.127503 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09\": container with ID starting with e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09 not found: ID does not exist" containerID="e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09" Oct 31 00:38:48.127608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.127527 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09"} err="failed to get container status \"e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09\": rpc error: code = NotFound desc = could not find container \"e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09\": container with ID starting with e7380eba94be7bf0191dc30c57df5c80b1f7d08515e53fe03b4127ec79623f09 not found: ID does not exist" Oct 31 00:38:48.135285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.135261 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh"] Oct 31 00:38:48.139496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:48.139477 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5fd7b8db97-gk8gh"] Oct 31 00:38:50.058755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:38:50.058720 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" path="/var/lib/kubelet/pods/c3fe98f1-5bfe-461e-bd81-e47643e7fd2d/volumes" Oct 31 00:39:13.084642 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:13.084603 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:39:32.188610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.188574 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk"] Oct 31 00:39:32.189189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.188834 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" podUID="d59eecd3-7750-4b71-ac82-dc9c85d74ea7" containerName="thanos-querier" containerID="cri-o://52155080d15f62e16239053ce07c663e368385353be893d219e0321b7fc2eb8e" gracePeriod=30 Oct 31 00:39:32.267684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.267640 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:39:32.268189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.267941 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="prometheus" containerID="cri-o://dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" gracePeriod=600 Oct 31 00:39:32.268189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.268006 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="config-reloader" containerID="cri-o://31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" gracePeriod=600 Oct 31 00:39:32.268189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.267961 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="thanos-sidecar" containerID="cri-o://2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" gracePeriod=600 Oct 31 00:39:32.317123 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.317098 5073 generic.go:356] "Generic (PLEG): container finished" podID="d59eecd3-7750-4b71-ac82-dc9c85d74ea7" containerID="52155080d15f62e16239053ce07c663e368385353be893d219e0321b7fc2eb8e" exitCode=0 Oct 31 00:39:32.317233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.317158 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" event={"ID":"d59eecd3-7750-4b71-ac82-dc9c85d74ea7","Type":"ContainerDied","Data":"52155080d15f62e16239053ce07c663e368385353be893d219e0321b7fc2eb8e"} Oct 31 00:39:32.499318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.499286 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw"] Oct 31 00:39:32.499638 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.499473 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" podUID="08e54ff8-faf4-4b68-a86c-fb14cde4cabb" containerName="ta-container" containerID="cri-o://ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c" gracePeriod=30 Oct 31 00:39:32.620540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.620488 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:39:32.655769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.655113 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv\" (UniqueName: \"kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv\") pod \"d59eecd3-7750-4b71-ac82-dc9c85d74ea7\" (UID: \"d59eecd3-7750-4b71-ac82-dc9c85d74ea7\") " Oct 31 00:39:32.660355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.659265 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv" pod "d59eecd3-7750-4b71-ac82-dc9c85d74ea7" (UID: "d59eecd3-7750-4b71-ac82-dc9c85d74ea7"). InnerVolumeSpecName "kube-api-access-wz5jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.670435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.667974 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:39:32.756090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756063 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756111 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756150 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756239 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756275 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756306 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756416 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756381 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756437 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756480 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756534 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756570 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756606 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.756728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756639 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"b2312322-6a7e-41b4-bf94-c989d27846b2\" (UID: \"b2312322-6a7e-41b4-bf94-c989d27846b2\") " Oct 31 00:39:32.757497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.756941 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz5jv\" (UniqueName: \"kubernetes.io/projected/d59eecd3-7750-4b71-ac82-dc9c85d74ea7-kube-api-access-wz5jv\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.757497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.757229 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.760559 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.760528 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:32.760670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.760617 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.760915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.760877 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.763944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.763916 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.764962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.764003 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.764962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.764417 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.764962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.764526 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.764962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.764740 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.766988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.766949 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "kube-api-access-gbk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.767134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.767110 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.773897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.773872 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:39:32.782550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.782528 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config" pod "b2312322-6a7e-41b4-bf94-c989d27846b2" (UID: "b2312322-6a7e-41b4-bf94-c989d27846b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.796315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.796292 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:39:32.858008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.857980 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal\" (UniqueName: \"kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal\") pod \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " Oct 31 00:39:32.858170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858036 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h\" (UniqueName: \"kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h\") pod \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " Oct 31 00:39:32.858170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858073 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert\") pod \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\" (UID: \"08e54ff8-faf4-4b68-a86c-fb14cde4cabb\") " Oct 31 00:39:32.858303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858291 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2312322-6a7e-41b4-bf94-c989d27846b2-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858312 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858330 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858359 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:39:32.858487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858374 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858389 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858397 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal" pod "08e54ff8-faf4-4b68-a86c-fb14cde4cabb" (UID: "08e54ff8-faf4-4b68-a86c-fb14cde4cabb"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.858487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858406 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858449 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858469 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858486 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbk9g\" (UniqueName: \"kubernetes.io/projected/b2312322-6a7e-41b4-bf94-c989d27846b2-kube-api-access-gbk9g\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858503 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858518 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2312322-6a7e-41b4-bf94-c989d27846b2-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.858730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.858534 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/b2312322-6a7e-41b4-bf94-c989d27846b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.860848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.860816 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert" pod "08e54ff8-faf4-4b68-a86c-fb14cde4cabb" (UID: "08e54ff8-faf4-4b68-a86c-fb14cde4cabb"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.860930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.860854 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h" pod "08e54ff8-faf4-4b68-a86c-fb14cde4cabb" (UID: "08e54ff8-faf4-4b68-a86c-fb14cde4cabb"). InnerVolumeSpecName "kube-api-access-jxb4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.869772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.869756 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:39:32.959337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.959286 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-b95813fe69a53e504648afb778a16429cd1cb9f758dcdaf86ddeca47cac76ae5\"" Oct 31 00:39:32.959337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.959308 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.959337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.959318 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxb4h\" (UniqueName: \"kubernetes.io/projected/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-kube-api-access-jxb4h\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.959337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:32.959329 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/08e54ff8-faf4-4b68-a86c-fb14cde4cabb-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:33.323954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.323928 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" Oct 31 00:39:33.324331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.323974 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk" event={"ID":"d59eecd3-7750-4b71-ac82-dc9c85d74ea7","Type":"ContainerDied","Data":"360aff1236d76779fbc4374581de089994086005cf81fa807cc3eefa8cd2c72c"} Oct 31 00:39:33.324331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.324023 5073 scope.go:122] "RemoveContainer" containerID="52155080d15f62e16239053ce07c663e368385353be893d219e0321b7fc2eb8e" Oct 31 00:39:33.326712 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326687 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerID="2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" exitCode=0 Oct 31 00:39:33.326856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326715 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerID="31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" exitCode=0 Oct 31 00:39:33.326856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326728 5073 generic.go:356] "Generic (PLEG): container finished" podID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerID="dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" exitCode=0 Oct 31 00:39:33.326856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326723 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerDied","Data":"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e"} Oct 31 00:39:33.326856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326758 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:39:33.326856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326836 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerDied","Data":"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26"} Oct 31 00:39:33.327088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326885 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerDied","Data":"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a"} Oct 31 00:39:33.327088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.326925 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"b2312322-6a7e-41b4-bf94-c989d27846b2","Type":"ContainerDied","Data":"c767db971282a878ada5a11a244b25eba17ed87972fd352a370dccc05e1c2366"} Oct 31 00:39:33.328597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.328571 5073 generic.go:356] "Generic (PLEG): container finished" podID="08e54ff8-faf4-4b68-a86c-fb14cde4cabb" containerID="ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c" exitCode=0 Oct 31 00:39:33.328699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.328618 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" Oct 31 00:39:33.328699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.328639 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" event={"ID":"08e54ff8-faf4-4b68-a86c-fb14cde4cabb","Type":"ContainerDied","Data":"ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c"} Oct 31 00:39:33.328699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.328668 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw" event={"ID":"08e54ff8-faf4-4b68-a86c-fb14cde4cabb","Type":"ContainerDied","Data":"d498c80fda7823efdb055061565d4f61bfab82c6cb102a75b75ba522d87a7213"} Oct 31 00:39:33.334679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.334227 5073 scope.go:122] "RemoveContainer" containerID="2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" Oct 31 00:39:33.346059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.346037 5073 scope.go:122] "RemoveContainer" containerID="31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" Oct 31 00:39:33.356189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.352634 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk"] Oct 31 00:39:33.356381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.356360 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-kzctk"] Oct 31 00:39:33.362500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.362483 5073 scope.go:122] "RemoveContainer" containerID="dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" Oct 31 00:39:33.370503 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.370482 5073 scope.go:122] "RemoveContainer" containerID="e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2" Oct 31 00:39:33.371138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.371116 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:39:33.376125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.376106 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:39:33.379311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.379295 5073 scope.go:122] "RemoveContainer" containerID="2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" Oct 31 00:39:33.379587 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:33.379570 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": container with ID starting with 2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e not found: ID does not exist" containerID="2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" Oct 31 00:39:33.379625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.379598 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e"} err="failed to get container status \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": rpc error: code = NotFound desc = could not find container \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": container with ID starting with 2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e not found: ID does not exist" Oct 31 00:39:33.379625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.379613 5073 scope.go:122] "RemoveContainer" containerID="31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" Oct 31 00:39:33.379895 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:33.379871 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": container with ID starting with 31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26 not found: ID does not exist" containerID="31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" Oct 31 00:39:33.379931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.379900 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26"} err="failed to get container status \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": rpc error: code = NotFound desc = could not find container \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": container with ID starting with 31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26 not found: ID does not exist" Oct 31 00:39:33.379931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.379914 5073 scope.go:122] "RemoveContainer" containerID="dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" Oct 31 00:39:33.380165 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:33.380146 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": container with ID starting with dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a not found: ID does not exist" containerID="dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" Oct 31 00:39:33.380202 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380175 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a"} err="failed to get container status \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": rpc error: code = NotFound desc = could not find container \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": container with ID starting with dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a not found: ID does not exist" Oct 31 00:39:33.380202 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380196 5073 scope.go:122] "RemoveContainer" containerID="e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2" Oct 31 00:39:33.380447 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:33.380430 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": container with ID starting with e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2 not found: ID does not exist" containerID="e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2" Oct 31 00:39:33.380509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380451 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2"} err="failed to get container status \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": rpc error: code = NotFound desc = could not find container \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": container with ID starting with e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2 not found: ID does not exist" Oct 31 00:39:33.380509 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380464 5073 scope.go:122] "RemoveContainer" containerID="2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" Oct 31 00:39:33.380682 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380661 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e"} err="failed to get container status \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": rpc error: code = NotFound desc = could not find container \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": container with ID starting with 2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e not found: ID does not exist" Oct 31 00:39:33.380723 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380683 5073 scope.go:122] "RemoveContainer" containerID="31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" Oct 31 00:39:33.380943 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380926 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26"} err="failed to get container status \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": rpc error: code = NotFound desc = could not find container \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": container with ID starting with 31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26 not found: ID does not exist" Oct 31 00:39:33.380991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.380944 5073 scope.go:122] "RemoveContainer" containerID="dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" Oct 31 00:39:33.381166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381147 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a"} err="failed to get container status \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": rpc error: code = NotFound desc = could not find container \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": container with ID starting with dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a not found: ID does not exist" Oct 31 00:39:33.381248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381167 5073 scope.go:122] "RemoveContainer" containerID="e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2" Oct 31 00:39:33.381425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381407 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2"} err="failed to get container status \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": rpc error: code = NotFound desc = could not find container \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": container with ID starting with e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2 not found: ID does not exist" Oct 31 00:39:33.381474 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381426 5073 scope.go:122] "RemoveContainer" containerID="2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e" Oct 31 00:39:33.381614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381594 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e"} err="failed to get container status \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": rpc error: code = NotFound desc = could not find container \"2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e\": container with ID starting with 2a129a6281795b78b67bf0bdba682d2481640db26f5c3470aa1c23528d57b82e not found: ID does not exist" Oct 31 00:39:33.381655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381615 5073 scope.go:122] "RemoveContainer" containerID="31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26" Oct 31 00:39:33.381829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381812 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26"} err="failed to get container status \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": rpc error: code = NotFound desc = could not find container \"31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26\": container with ID starting with 31c1b0af03cac03b3269cfae2d120c231628799a1d6e00f8ad9bd46f7e34ad26 not found: ID does not exist" Oct 31 00:39:33.381829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.381829 5073 scope.go:122] "RemoveContainer" containerID="dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a" Oct 31 00:39:33.382022 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.382005 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a"} err="failed to get container status \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": rpc error: code = NotFound desc = could not find container \"dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a\": container with ID starting with dc2b41ea75bf2430daeb25031245e2ef2be21364db0ad17110bac139990bdf1a not found: ID does not exist" Oct 31 00:39:33.382061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.382023 5073 scope.go:122] "RemoveContainer" containerID="e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2" Oct 31 00:39:33.382214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.382199 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2"} err="failed to get container status \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": rpc error: code = NotFound desc = could not find container \"e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2\": container with ID starting with e825c273b49300dce7469edd19321d554b2104d801b65d14064db3ab6bf2bee2 not found: ID does not exist" Oct 31 00:39:33.382274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.382215 5073 scope.go:122] "RemoveContainer" containerID="ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c" Oct 31 00:39:33.386963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.386946 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw"] Oct 31 00:39:33.390028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.390013 5073 scope.go:122] "RemoveContainer" containerID="ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c" Oct 31 00:39:33.390400 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:33.390381 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c\": container with ID starting with ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c not found: ID does not exist" containerID="ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c" Oct 31 00:39:33.390467 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.390404 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c"} err="failed to get container status \"ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c\": rpc error: code = NotFound desc = could not find container \"ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c\": container with ID starting with ae2140bf83374b458e60bf54d0af3460422f69281f4a55aab784cc3b2ed1ce8c not found: ID does not exist" Oct 31 00:39:33.390995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:33.390977 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6ffc7c9b4f-frmvw"] Oct 31 00:39:34.057724 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.057696 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e54ff8-faf4-4b68-a86c-fb14cde4cabb" path="/var/lib/kubelet/pods/08e54ff8-faf4-4b68-a86c-fb14cde4cabb/volumes" Oct 31 00:39:34.058178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.058164 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" path="/var/lib/kubelet/pods/b2312322-6a7e-41b4-bf94-c989d27846b2/volumes" Oct 31 00:39:34.058576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.058564 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59eecd3-7750-4b71-ac82-dc9c85d74ea7" path="/var/lib/kubelet/pods/d59eecd3-7750-4b71-ac82-dc9c85d74ea7/volumes" Oct 31 00:39:34.388327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388257 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388575 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="config-reloader" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388585 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="config-reloader" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388602 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="init-config-reloader" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388607 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="init-config-reloader" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388615 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="thanos-sidecar" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388620 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="thanos-sidecar" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388632 5073 cpu_manager.go:414] "containerMap: removing container" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="prometheus" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388637 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="prometheus" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388648 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d59eecd3-7750-4b71-ac82-dc9c85d74ea7" containerName="thanos-querier" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388653 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59eecd3-7750-4b71-ac82-dc9c85d74ea7" containerName="thanos-querier" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388664 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" containerName="ta-container" Oct 31 00:39:34.388664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388668 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" containerName="ta-container" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388679 5073 cpu_manager.go:414] "containerMap: removing container" podUID="08e54ff8-faf4-4b68-a86c-fb14cde4cabb" containerName="ta-container" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388684 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e54ff8-faf4-4b68-a86c-fb14cde4cabb" containerName="ta-container" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388731 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59eecd3-7750-4b71-ac82-dc9c85d74ea7" containerName="thanos-querier" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388738 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fe98f1-5bfe-461e-bd81-e47643e7fd2d" containerName="ta-container" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388745 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e54ff8-faf4-4b68-a86c-fb14cde4cabb" containerName="ta-container" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388749 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="config-reloader" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388756 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="thanos-sidecar" Oct 31 00:39:34.389079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.388762 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2312322-6a7e-41b4-bf94-c989d27846b2" containerName="prometheus" Oct 31 00:39:34.393453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.393434 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.400686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.399593 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:39:34.400686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.400210 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:39:34.400987 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.400967 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:39:34.401259 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.401237 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:39:34.469819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.469771 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswwf\" (UniqueName: \"kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf\") pod \"seaweedfs\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.469955 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.469873 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data\") pod \"seaweedfs\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.570826 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.570798 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswwf\" (UniqueName: \"kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf\") pod \"seaweedfs\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.570928 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.570836 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data\") pod \"seaweedfs\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.571132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.571117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data\") pod \"seaweedfs\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.583760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.583737 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswwf\" (UniqueName: \"kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf\") pod \"seaweedfs\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.706843 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.706766 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:34.978490 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:34.978428 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:39:34.981741 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:39:34.981711 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa849695_ec53_4f11_b357_74e71e5afc91.slice/crio-485726ff70ba33189e7614253c83c2023849da73ad9a3cbe9bdd70b5d2454ecd WatchSource:0}: Error finding container 485726ff70ba33189e7614253c83c2023849da73ad9a3cbe9bdd70b5d2454ecd: Status 404 returned error can't find the container with id 485726ff70ba33189e7614253c83c2023849da73ad9a3cbe9bdd70b5d2454ecd Oct 31 00:39:35.342100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:35.342068 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"aa849695-ec53-4f11-b357-74e71e5afc91","Type":"ContainerStarted","Data":"a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7"} Oct 31 00:39:35.342284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:35.342111 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"aa849695-ec53-4f11-b357-74e71e5afc91","Type":"ContainerStarted","Data":"485726ff70ba33189e7614253c83c2023849da73ad9a3cbe9bdd70b5d2454ecd"} Oct 31 00:39:35.342284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:35.342196 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:35.367973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:35.367928 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/seaweedfs" podStartSLOduration=1.367915202 podStartE2EDuration="1.367915202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:35.367259332 +0000 UTC m=+2459.428034029" watchObservedRunningTime="2026-10-31 00:39:35.367915202 +0000 UTC m=+2459.428689897" Oct 31 00:39:37.145547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:37.145518 5073 scope.go:122] "RemoveContainer" containerID="f09d65b04557b2865188a43fea3bda05254efaf5c698fbffc68c9ecacd63866d" Oct 31 00:39:43.087854 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:39:43.087820 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:39:46.349513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:46.349431 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:39:53.802217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.802186 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz"] Oct 31 00:39:53.807323 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.807304 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.812726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.812703 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 31 00:39:53.812726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.812716 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:39:53.813013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.812880 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-5frhp\"" Oct 31 00:39:53.813013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.812882 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 31 00:39:53.818767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.818742 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz"] Oct 31 00:39:53.896520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.896491 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:39:53.896520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.896492 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 31 00:39:53.896520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.896504 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 31 00:39:53.917981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.917958 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.917988 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.918007 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrhm\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.918057 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.918117 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.918163 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.918185 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:53.918320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:53.918207 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrhm\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019474 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019496 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019522 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019542 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019854 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.019984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.019960 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.020046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.020006 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.020277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.020248 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.020372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.020351 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.020647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.020626 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.022806 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.022753 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.022918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.022855 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.022918 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.022891 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.023031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.022955 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.033439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.033412 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrhm\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.134205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.134129 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:39:54.416679 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.416653 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz"] Oct 31 00:39:54.419378 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:39:54.419351 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2740588_0d1b_47d6_a95c_1ea7b0332673.slice/crio-b9eb93df819f01d2675d649e192aa9c3680f3b151c842874de53db0c65ae78d3 WatchSource:0}: Error finding container b9eb93df819f01d2675d649e192aa9c3680f3b151c842874de53db0c65ae78d3: Status 404 returned error can't find the container with id b9eb93df819f01d2675d649e192aa9c3680f3b151c842874de53db0c65ae78d3 Oct 31 00:39:54.421703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.421684 5073 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:39:54.427939 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:54.427919 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" event={"ID":"f2740588-0d1b-47d6-a95c-1ea7b0332673","Type":"ContainerStarted","Data":"b9eb93df819f01d2675d649e192aa9c3680f3b151c842874de53db0c65ae78d3"} Oct 31 00:39:56.610921 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.610887 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:39:56.614494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.614474 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.618671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.618648 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-usage-logs-loki-secret\"" Oct 31 00:39:56.619738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.619719 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 31 00:39:56.619878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.619778 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-grpc\"" Oct 31 00:39:56.619878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.619836 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-http\"" Oct 31 00:39:56.619964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.619911 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-btcx5\"" Oct 31 00:39:56.627410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.627390 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:39:56.664553 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.664528 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:39:56.668026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.668006 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.679683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.679662 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:39:56.718849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.718813 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 31 00:39:56.718973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.718879 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 31 00:39:56.741560 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741530 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.741560 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741560 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.741753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741581 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbl8\" (UniqueName: \"kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.741753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741663 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.741753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741696 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.741753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741729 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.741990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741837 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22w8r\" (UniqueName: \"kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.741990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741887 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.741990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741906 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.741990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741931 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.741990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741947 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.741990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.741962 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.842709 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842666 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.842878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842732 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.842878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842773 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.842878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842820 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22w8r\" (UniqueName: \"kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.843021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842877 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.843021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842904 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.843021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842931 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.843021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842957 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.843021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.842981 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.843238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.843041 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.843238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.843072 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.843238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.843102 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbl8\" (UniqueName: \"kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.844614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.844369 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.844614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.844479 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.844614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.844530 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.845139 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.845059 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.846675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.846651 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.846810 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.846765 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.847102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.847082 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.847174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.847125 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.847224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.847194 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.847986 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.847968 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:56.856127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.856101 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22w8r\" (UniqueName: \"kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:39:56.856986 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:56.856966 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbl8\" (UniqueName: \"kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:39:58.449265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:39:58.449227 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" event={"ID":"f2740588-0d1b-47d6-a95c-1ea7b0332673","Type":"ContainerStarted","Data":"fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161"} Oct 31 00:40:00.461099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:00.461001 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" event={"ID":"f2740588-0d1b-47d6-a95c-1ea7b0332673","Type":"ContainerStarted","Data":"40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03"} Oct 31 00:40:00.461495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:00.461157 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:40:00.461495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:00.461198 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:40:00.467263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:00.467242 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:40:00.470755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:00.470737 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:40:00.485959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:00.485919 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" podStartSLOduration=1.958045046 podStartE2EDuration="7.485905282s" totalImagesPullingTime="5.527860236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:00.484966121 +0000 UTC m=+2484.545740819" watchObservedRunningTime="2026-10-31 00:40:00.485905282 +0000 UTC m=+2484.546680155" Oct 31 00:40:06.420239 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.420189 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.420824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.420284 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.420824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.420324 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.428881 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.428858 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.429002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.428892 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.429002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.428920 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.521195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.521164 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.521353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.521225 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.521353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.521266 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.521353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.521312 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.521532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.521372 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.521532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.521374 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.524876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.524857 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") DevicePath \"csi-0e3ae4b7a4a45b7fac2e73d30f486b2b5989e25d15f0b8f632bc09c0fb516836\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.525004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.524877 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") DevicePath \"csi-e6fd39fc46b3d77f941fdbebf9769b8424ddd772f87e48a52f180c692830420f\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.525004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.524859 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") DevicePath \"csi-70efa0f79eac0e40982684e61df4f3833412c2cd0157e43aa1a3aa8d5e4dbb9e\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.742701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.742664 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7c9d44c2ac5b185f6db2eab8b6cc11f2151bd1a519bd148465043863d898a7a3/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.744653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.744626 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ce8a44ea6fd624fabb855523df239646d3dcc3258d2c5f84025b15690a2664c7/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.746007 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.745984 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fdf3d44672bff7110481cf42cbe6e99aa39e5c8305a671c1179f840d2ab17bcc/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.757763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.757737 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.758963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.758943 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:06.760723 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.760702 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.829551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.829526 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:06.881092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:06.881069 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:07.107442 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:07.107411 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:40:07.110632 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:40:07.110601 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0738c47_24ab_414d_99b7_181edf6477ff.slice/crio-524850d0d79ae9b0cb526eff7f04e417e12ba2f63c0b23f30b5885a2a6e8db08 WatchSource:0}: Error finding container 524850d0d79ae9b0cb526eff7f04e417e12ba2f63c0b23f30b5885a2a6e8db08: Status 404 returned error can't find the container with id 524850d0d79ae9b0cb526eff7f04e417e12ba2f63c0b23f30b5885a2a6e8db08 Oct 31 00:40:07.165050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:07.165022 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:40:07.167441 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:40:07.167414 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebfefa87_b20a_4dc0_a543_a647be527cd5.slice/crio-c91a7fe2f906f44e1d9ac9c62ccb9f21220ce5f7473c7ab4df0f7608afdf39fa WatchSource:0}: Error finding container c91a7fe2f906f44e1d9ac9c62ccb9f21220ce5f7473c7ab4df0f7608afdf39fa: Status 404 returned error can't find the container with id c91a7fe2f906f44e1d9ac9c62ccb9f21220ce5f7473c7ab4df0f7608afdf39fa Oct 31 00:40:07.491793 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:07.491744 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"e0738c47-24ab-414d-99b7-181edf6477ff","Type":"ContainerStarted","Data":"524850d0d79ae9b0cb526eff7f04e417e12ba2f63c0b23f30b5885a2a6e8db08"} Oct 31 00:40:07.493075 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:07.493047 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"ebfefa87-b20a-4dc0-a543-a647be527cd5","Type":"ContainerStarted","Data":"c91a7fe2f906f44e1d9ac9c62ccb9f21220ce5f7473c7ab4df0f7608afdf39fa"} Oct 31 00:40:10.511631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:10.511589 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"e0738c47-24ab-414d-99b7-181edf6477ff","Type":"ContainerStarted","Data":"dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8"} Oct 31 00:40:10.512116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:10.511640 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:40:10.513270 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:10.513245 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"ebfefa87-b20a-4dc0-a543-a647be527cd5","Type":"ContainerStarted","Data":"df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b"} Oct 31 00:40:10.513365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:10.513334 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:10.533258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:10.533211 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podStartSLOduration=14.499636989 podStartE2EDuration="17.533198242s" totalImagesPullingTime="3.033561253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:10.532740993 +0000 UTC m=+2494.593515688" watchObservedRunningTime="2026-10-31 00:40:10.533198242 +0000 UTC m=+2494.593972938" Oct 31 00:40:10.556920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:10.556881 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podStartSLOduration=14.710309295 podStartE2EDuration="17.556869119s" totalImagesPullingTime="2.846559824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:10.55570029 +0000 UTC m=+2494.616474988" watchObservedRunningTime="2026-10-31 00:40:10.556869119 +0000 UTC m=+2494.617643814" Oct 31 00:40:13.084581 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:40:13.084546 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:40:31.531003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:31.530958 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:31.531516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:31.531497 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:40:41.529529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:41.529493 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:40:43.084923 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:40:43.084883 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:40:51.529632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:40:51.529588 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:01.529175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:01.529136 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:11.529021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:11.528990 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:41:13.084418 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:41:13.084383 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:41:27.681740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.681658 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59"] Oct 31 00:41:27.685464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.685438 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.693810 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.693750 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 31 00:41:27.694745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.694723 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 31 00:41:27.694893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.694843 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\"" Oct 31 00:41:27.694960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.694894 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-ww784\"" Oct 31 00:41:27.694960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.694932 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\"" Oct 31 00:41:27.696174 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.696150 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59"] Oct 31 00:41:27.752927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.752894 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.753093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.752988 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.753093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.753019 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.753093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.753072 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbqx\" (UniqueName: \"kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.753275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.753185 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.852691 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.852652 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp"] Oct 31 00:41:27.854337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.854305 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.854440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.854404 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.854440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.854433 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.854510 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.854487 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbqx\" (UniqueName: \"kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.854568 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.854554 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.855216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.855184 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.855422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.855395 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.856358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.856334 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:27.857444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.857423 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.857721 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.857700 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.867478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.867454 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp"] Oct 31 00:41:27.869738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.869705 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbqx\" (UniqueName: \"kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx\") pod \"data-science-lokistack-distributor-b854ccfb4-gjn59\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:27.894031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.894007 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\"" Oct 31 00:41:27.894134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.894043 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\"" Oct 31 00:41:27.955054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.954985 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:27.955054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.955021 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:27.955258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.955151 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnncd\" (UniqueName: \"kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:27.955319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.955268 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:27.955381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.955335 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:27.976770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.976739 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg"] Oct 31 00:41:27.980358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.980338 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:27.994119 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.994096 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 31 00:41:27.994653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.994608 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 31 00:41:27.994775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.994711 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 31 00:41:27.994890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.994825 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-whtcj\"" Oct 31 00:41:27.995432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.995408 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg"] Oct 31 00:41:27.999022 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:27.999000 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056338 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056480 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056522 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056554 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056586 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056610 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056640 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056682 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056738 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056775 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056833 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056873 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmdn\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.058177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.056957 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnncd\" (UniqueName: \"kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.058959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.058454 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.058959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.058468 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.060715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.060696 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.061496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.061474 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.071088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.071064 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 31 00:41:28.071214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.071093 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 31 00:41:28.077478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.077457 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnncd\" (UniqueName: \"kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd\") pod \"data-science-lokistack-query-frontend-5cddddc95c-69hxp\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.157850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157776 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmdn\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157873 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157918 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157939 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157956 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157978 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.157998 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.158319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.158043 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.159015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.158949 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.159015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.158983 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.159166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.159037 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.160595 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.160572 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.161076 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.161057 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.161177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.161157 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.161594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.161576 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.170121 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.170099 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmdn\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn\") pod \"data-science-lokistack-gateway-66cfb97849-cpnvg\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.192031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.192009 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.276263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.276237 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59"] Oct 31 00:41:28.279844 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:41:28.279814 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5cc166c_dd4f_42a0_b3cc_01296493c45f.slice/crio-4874f228470c7a203b6e0e6abb91102fc31d21788404c6a335a4f3d4db58739e WatchSource:0}: Error finding container 4874f228470c7a203b6e0e6abb91102fc31d21788404c6a335a4f3d4db58739e: Status 404 returned error can't find the container with id 4874f228470c7a203b6e0e6abb91102fc31d21788404c6a335a4f3d4db58739e Oct 31 00:41:28.295088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.295066 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.470202 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.470162 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp"] Oct 31 00:41:28.473088 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:41:28.473054 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfca95d77_aeb8_48a0_a16a_3decd9bdbbe1.slice/crio-acec0fbde816f0615925f02b93ce51ea3e476d93e02f8ab870327de51fef8fc7 WatchSource:0}: Error finding container acec0fbde816f0615925f02b93ce51ea3e476d93e02f8ab870327de51fef8fc7: Status 404 returned error can't find the container with id acec0fbde816f0615925f02b93ce51ea3e476d93e02f8ab870327de51fef8fc7 Oct 31 00:41:28.586830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.586519 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg"] Oct 31 00:41:28.589462 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:41:28.589426 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41c54ace_b3d3_4297_922d_9082e4584ae6.slice/crio-b9e85e3aee0b9487545db80cb4d9e3ff440eb5a4dbcdb0f6900b8ffa6430cc6b WatchSource:0}: Error finding container b9e85e3aee0b9487545db80cb4d9e3ff440eb5a4dbcdb0f6900b8ffa6430cc6b: Status 404 returned error can't find the container with id b9e85e3aee0b9487545db80cb4d9e3ff440eb5a4dbcdb0f6900b8ffa6430cc6b Oct 31 00:41:28.873316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.873245 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" event={"ID":"41c54ace-b3d3-4297-922d-9082e4584ae6","Type":"ContainerStarted","Data":"8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441"} Oct 31 00:41:28.873316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.873296 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.874054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.873314 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" event={"ID":"41c54ace-b3d3-4297-922d-9082e4584ae6","Type":"ContainerStarted","Data":"4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a"} Oct 31 00:41:28.874054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.873336 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.874054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.873348 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" event={"ID":"41c54ace-b3d3-4297-922d-9082e4584ae6","Type":"ContainerStarted","Data":"b9e85e3aee0b9487545db80cb4d9e3ff440eb5a4dbcdb0f6900b8ffa6430cc6b"} Oct 31 00:41:28.875302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.875277 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" event={"ID":"a5cc166c-dd4f-42a0-b3cc-01296493c45f","Type":"ContainerStarted","Data":"9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99"} Oct 31 00:41:28.875430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.875333 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:28.875430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.875351 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" event={"ID":"a5cc166c-dd4f-42a0-b3cc-01296493c45f","Type":"ContainerStarted","Data":"4874f228470c7a203b6e0e6abb91102fc31d21788404c6a335a4f3d4db58739e"} Oct 31 00:41:28.877239 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.877220 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" event={"ID":"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1","Type":"ContainerStarted","Data":"8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb"} Oct 31 00:41:28.877340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.877250 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" event={"ID":"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1","Type":"ContainerStarted","Data":"acec0fbde816f0615925f02b93ce51ea3e476d93e02f8ab870327de51fef8fc7"} Oct 31 00:41:28.877340 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.877324 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:41:28.880473 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.880456 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.883033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.883017 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:41:28.898629 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.898587 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" podStartSLOduration=1.898575212 podStartE2EDuration="1.898575212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:28.896777275 +0000 UTC m=+2572.957551971" watchObservedRunningTime="2026-10-31 00:41:28.898575212 +0000 UTC m=+2572.959349899" Oct 31 00:41:28.915663 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.915622 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" podStartSLOduration=1.915611332 podStartE2EDuration="1.915611332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:28.915161673 +0000 UTC m=+2572.975936369" watchObservedRunningTime="2026-10-31 00:41:28.915611332 +0000 UTC m=+2572.976386024" Oct 31 00:41:28.959699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:28.959650 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" podStartSLOduration=1.959633806 podStartE2EDuration="1.959633806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:28.958610731 +0000 UTC m=+2573.019385441" watchObservedRunningTime="2026-10-31 00:41:28.959633806 +0000 UTC m=+2573.020408502" Oct 31 00:41:30.765380 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.765338 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 31 00:41:30.768960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.768942 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.772893 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.772766 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-logs-loki-secret\"" Oct 31 00:41:30.773476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.773460 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\"" Oct 31 00:41:30.773542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.773510 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\"" Oct 31 00:41:30.780774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.780753 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 31 00:41:30.885189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.885160 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-test-logs-loki-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.885312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.885212 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.885312 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.885298 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.885435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.885332 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.885435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.885353 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.885518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.885448 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272bh\" (UniqueName: \"kubernetes.io/projected/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-kube-api-access-272bh\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.913355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.913324 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 31 00:41:30.916620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.916605 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:30.928122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.928099 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 31 00:41:30.973536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.973511 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 31 00:41:30.973654 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.973553 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 31 00:41:30.986083 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.986058 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272bh\" (UniqueName: \"kubernetes.io/projected/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-kube-api-access-272bh\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.986131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.986108 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-test-logs-loki-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.986163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.986138 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.986201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.986165 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.986201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.986190 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.986276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.986235 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.987071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.987048 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.987389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.987365 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-config\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.988954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.988924 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.989048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.988933 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.989090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.989073 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-test-logs-loki-secret\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:30.998627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:30.998609 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-272bh\" (UniqueName: \"kubernetes.io/projected/c57125c0-82f4-40e5-b8ea-aeba9fdab92c-kube-api-access-272bh\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:31.086975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.086922 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.086975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.086954 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.087150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.086973 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.087211 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.087141 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlp8n\" (UniqueName: \"kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.087345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.087322 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.087449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.087434 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.188235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.188200 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.188350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.188271 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.188350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.188310 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.188459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.188353 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.188459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.188380 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.188700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.188676 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlp8n\" (UniqueName: \"kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.189284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.189257 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.189409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.189387 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.190906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.190882 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.190984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.190928 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.191218 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.191197 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.201853 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.201832 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlp8n\" (UniqueName: \"kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:31.720573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.720543 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bm4v9"] Oct 31 00:41:31.725435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.725413 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.737258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.737229 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bm4v9"] Oct 31 00:41:31.775305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.775282 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 31 00:41:31.775680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.775484 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 31 00:41:31.775680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.775516 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-fkhp6\"" Oct 31 00:41:31.792764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.792742 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bm4v9"] Oct 31 00:41:31.792999 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:41:31.792977 5073 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[config config-openshift-service-cacrt data-science-cluster-log-forwarder-collector-token datadir entrypoint kube-api-access-v94k4 metrics sa-token tmp trusted-ca varlogpods], unattached volumes=[], failed to process volumes=[config config-openshift-service-cacrt data-science-cluster-log-forwarder-collector-token datadir entrypoint kube-api-access-v94k4 metrics sa-token tmp trusted-ca varlogpods]: context canceled" pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" podUID="20115130-84f8-4e46-a5c9-c906f0bb1035" Oct 31 00:41:31.813296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.813269 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 31 00:41:31.813412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.813348 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 31 00:41:31.813511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.813494 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 31 00:41:31.815834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.815816 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 31 00:41:31.892101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.892080 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.895731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.895711 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.895857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.895744 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.895857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.895797 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.895949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.895932 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v94k4\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.895963 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.895990 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.896014 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896116 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.896084 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.896180 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.896208 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.896321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.896302 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.898798 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.898764 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997193 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997237 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997255 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997371 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997438 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v94k4\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997496 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997472 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997507 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997531 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997565 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997580 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997648 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997656 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.997917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.997689 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.998158 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.998135 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.998441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.998419 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.998493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.998418 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:31.998650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.998633 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.000000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:31.999979 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.000286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.000266 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.000438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.000421 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.009464 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.009439 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v94k4\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.009582 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.009565 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token\") pod \"data-science-cluster-log-forwarder-bm4v9\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.098372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098346 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098401 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098436 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp\" (UniqueName: \"kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098462 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098497 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098618 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098672 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098701 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:41:32.098745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098706 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.098942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098814 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.099010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098933 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.099010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098973 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:41:32.099010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.098991 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.099167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099080 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.099167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099146 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.099342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099318 5073 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-datadir\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.099441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099345 5073 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-entrypoint\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.099441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099363 5073 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config-openshift-service-cacrt\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.099441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099378 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.099441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099391 5073 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/20115130-84f8-4e46-a5c9-c906f0bb1035-varlogpods\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.099441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.099406 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20115130-84f8-4e46-a5c9-c906f0bb1035-trusted-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.100597 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.100563 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:32.100681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.100577 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:32.100984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.100964 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:32.200168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.200142 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.200267 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.200242 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token\") pod \"20115130-84f8-4e46-a5c9-c906f0bb1035\" (UID: \"20115130-84f8-4e46-a5c9-c906f0bb1035\") " Oct 31 00:41:32.200415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.200402 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.200494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.200416 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20115130-84f8-4e46-a5c9-c906f0bb1035-tmp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.200494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.200426 5073 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/20115130-84f8-4e46-a5c9-c906f0bb1035-metrics\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.202133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.202114 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "kube-api-access-v94k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:32.202275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.202246 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token" pod "20115130-84f8-4e46-a5c9-c906f0bb1035" (UID: "20115130-84f8-4e46-a5c9-c906f0bb1035"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:32.301452 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.301426 5073 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-sa-token\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.301543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.301460 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v94k4\" (UniqueName: \"kubernetes.io/projected/20115130-84f8-4e46-a5c9-c906f0bb1035-kube-api-access-v94k4\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.896871 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.896801 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bm4v9" Oct 31 00:41:32.934736 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.934712 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bm4v9"] Oct 31 00:41:32.939720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.939699 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bm4v9"] Oct 31 00:41:32.954882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.954862 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-9pnbq"] Oct 31 00:41:32.959412 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.959396 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:32.963379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.963343 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 31 00:41:32.963495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.963478 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 31 00:41:32.963550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.963494 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 31 00:41:32.963550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.963495 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 31 00:41:32.963550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.963482 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 31 00:41:32.963550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.963544 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-fkhp6\"" Oct 31 00:41:32.966501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.966481 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 31 00:41:32.970941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:32.970920 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-9pnbq"] Oct 31 00:41:33.006948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.006927 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.006977 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007045 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007060 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007121 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007152 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007177 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007195 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007285 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007330 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.007436 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.007414 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nz2\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108039 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108083 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108120 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nz2\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108142 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108176 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108211 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108235 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108253 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108359 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108393 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108419 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108468 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.108730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.108585 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.109037 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.109010 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.109171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.109148 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.109256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.109220 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.109584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.109565 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.110579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.110560 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.110769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.110746 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.110829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.110770 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.121150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.121111 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nz2\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.121242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.121197 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token\") pod \"data-science-cluster-log-forwarder-9pnbq\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.274179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.274149 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:41:33.555894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.555873 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-9pnbq"] Oct 31 00:41:33.558646 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:41:33.558612 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4c42e2_37c4_4720_b729_4d45fcd67e1e.slice/crio-fcea90a13b09a07209336af11e6c2e9e9d474dcbf2566eec505c76d728f933de WatchSource:0}: Error finding container fcea90a13b09a07209336af11e6c2e9e9d474dcbf2566eec505c76d728f933de: Status 404 returned error can't find the container with id fcea90a13b09a07209336af11e6c2e9e9d474dcbf2566eec505c76d728f933de Oct 31 00:41:33.902283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:33.902207 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" event={"ID":"ae4c42e2-37c4-4720-b729-4d45fcd67e1e","Type":"ContainerStarted","Data":"fcea90a13b09a07209336af11e6c2e9e9d474dcbf2566eec505c76d728f933de"} Oct 31 00:41:34.059206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:34.059172 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20115130-84f8-4e46-a5c9-c906f0bb1035" path="/var/lib/kubelet/pods/20115130-84f8-4e46-a5c9-c906f0bb1035/volumes" Oct 31 00:41:37.186073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:37.186038 5073 scope.go:122] "RemoveContainer" containerID="709a0bba1c9854bd109ce29919a4da4547ac5ae6d525beb2b02f81432b1294a6" Oct 31 00:41:38.349264 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.349222 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.349685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.349322 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.349685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.349367 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.357272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.357179 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.357272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.357241 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.357272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.357258 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.450062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.449980 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.450062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.450050 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.450314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.450082 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.450314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.450146 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.450314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.450199 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.450314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.450206 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.453638 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.453617 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") DevicePath \"csi-6bec46f7ab78d6eb2a3ec126a7d6f2b1451c9ae8c05c7047d0d0c13ce0476469\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.453746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.453641 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") DevicePath \"csi-f3e313e21636c4c870fea1828de39353c6c1dea16aca7208a4e2c407b638a05a\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.453746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.453641 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") DevicePath \"csi-0a6bb46aac83074ea2c3e67a24b3628e870010d62628478534ad1be69cc731cb\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.683895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.683858 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/72d0d285f5ca9f756240ae49ec30d6069474ce4edb766ef0e88834d689b76186/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.686217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.686170 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0fddb5f40dc01fb82a514c1e121674f8d5f210f903954877488e75adc2f443fc/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.686855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.686714 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/43d92cade942035a915164a007f60f3e1dda6c02885b8a62b908af169343a0ff/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.704217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.704160 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-44ea1998-9069-482d-9abd-b316157b8e05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d555dc02244d95e8\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.706173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.706154 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ddaaae30-f254-4538-bef8-d7c41bb1bfd4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04d31e4c1f50ee864\") pod \"data-science-lokistack-ingester-0\" (UID: \"c57125c0-82f4-40e5-b8ea-aeba9fdab92c\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:38.708677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.708631 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.727774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.727751 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:38.882040 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:38.882008 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:39.012162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.012136 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 31 00:41:39.013936 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:41:39.013900 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24e241e5_9087_4179_97c9_4707d39d7718.slice/crio-1e856be304f7681623e2d1e4386e5fc4d35845b506978634bf42c1182a04e984 WatchSource:0}: Error finding container 1e856be304f7681623e2d1e4386e5fc4d35845b506978634bf42c1182a04e984: Status 404 returned error can't find the container with id 1e856be304f7681623e2d1e4386e5fc4d35845b506978634bf42c1182a04e984 Oct 31 00:41:39.173891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.173863 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 31 00:41:39.178564 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:41:39.178526 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc57125c0_82f4_40e5_b8ea_aeba9fdab92c.slice/crio-ebca1ea3aa58288e50a21b2b31dc49d7e3e108c5e304d7cbd5be87ed6cdd8c42 WatchSource:0}: Error finding container ebca1ea3aa58288e50a21b2b31dc49d7e3e108c5e304d7cbd5be87ed6cdd8c42: Status 404 returned error can't find the container with id ebca1ea3aa58288e50a21b2b31dc49d7e3e108c5e304d7cbd5be87ed6cdd8c42 Oct 31 00:41:39.934705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.934667 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"24e241e5-9087-4179-97c9-4707d39d7718","Type":"ContainerStarted","Data":"bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41"} Oct 31 00:41:39.935185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.934717 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"24e241e5-9087-4179-97c9-4707d39d7718","Type":"ContainerStarted","Data":"1e856be304f7681623e2d1e4386e5fc4d35845b506978634bf42c1182a04e984"} Oct 31 00:41:39.935185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.934895 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:41:39.936634 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.936602 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"c57125c0-82f4-40e5-b8ea-aeba9fdab92c","Type":"ContainerStarted","Data":"5b812127c55597e144e7db8e78a5421d55819cdc419d4d956152c22ffee5b811"} Oct 31 00:41:39.936769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.936645 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" event={"ID":"c57125c0-82f4-40e5-b8ea-aeba9fdab92c","Type":"ContainerStarted","Data":"ebca1ea3aa58288e50a21b2b31dc49d7e3e108c5e304d7cbd5be87ed6cdd8c42"} Oct 31 00:41:39.936769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.936727 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:41:39.957199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.957148 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=12.957137345 podStartE2EDuration="12.957137345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:39.955933386 +0000 UTC m=+2584.016708091" watchObservedRunningTime="2026-10-31 00:41:39.957137345 +0000 UTC m=+2584.017912041" Oct 31 00:41:39.978011 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:39.977962 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podStartSLOduration=12.977946296 podStartE2EDuration="12.977946296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:39.97734703 +0000 UTC m=+2584.038121727" watchObservedRunningTime="2026-10-31 00:41:39.977946296 +0000 UTC m=+2584.038720989" Oct 31 00:41:40.944828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:40.944771 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" event={"ID":"ae4c42e2-37c4-4720-b729-4d45fcd67e1e","Type":"ContainerStarted","Data":"d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852"} Oct 31 00:41:40.970641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:40.970595 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" podStartSLOduration=2.41935863 podStartE2EDuration="8.970582432s" totalImagesPullingTime="6.551223802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:40.967849191 +0000 UTC m=+2585.028623905" watchObservedRunningTime="2026-10-31 00:41:40.970582432 +0000 UTC m=+2585.031357296" Oct 31 00:41:43.085256 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:41:43.085220 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:41:49.888016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:49.887986 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:41:49.888573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:41:49.888380 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:42:00.950007 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:00.949963 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:00.950551 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:00.950530 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:42:10.949758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:10.949715 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:13.085254 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:13.085219 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:42:20.950404 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:20.950355 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:30.949876 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:30.949843 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:42:43.085355 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:43.085317 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:42:45.493622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.493591 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 31 00:42:45.494515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.494484 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="24e241e5-9087-4179-97c9-4707d39d7718" containerName="loki-index-gateway" containerID="cri-o://bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41" gracePeriod=30 Oct 31 00:42:45.545088 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.545053 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-0"] Oct 31 00:42:45.545924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.545720 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" containerID="cri-o://5b812127c55597e144e7db8e78a5421d55819cdc419d4d956152c22ffee5b811" gracePeriod=600 Oct 31 00:42:45.563585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.562266 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp"] Oct 31 00:42:45.563585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.562483 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" podUID="fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" containerName="loki-query-frontend" containerID="cri-o://8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb" gracePeriod=30 Oct 31 00:42:45.568372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.568344 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg"] Oct 31 00:42:45.568864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.568828 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="opa" containerID="cri-o://8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441" gracePeriod=30 Oct 31 00:42:45.569411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.569024 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="gateway" containerID="cri-o://4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a" gracePeriod=30 Oct 31 00:42:45.591758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.591701 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-9pnbq"] Oct 31 00:42:45.591978 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.591952 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" podUID="ae4c42e2-37c4-4720-b729-4d45fcd67e1e" containerName="collector" containerID="cri-o://d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852" gracePeriod=10 Oct 31 00:42:45.611975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.611944 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59"] Oct 31 00:42:45.612205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.612179 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" podUID="a5cc166c-dd4f-42a0-b3cc-01296493c45f" containerName="loki-distributor" containerID="cri-o://9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99" gracePeriod=30 Oct 31 00:42:45.923033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.923004 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:42:45.926002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.925980 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:42:45.929517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:45.929497 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:42:46.042299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042272 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.042447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042315 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042345 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.042447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042370 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.042447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042392 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config\") pod \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " Oct 31 00:42:46.042447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042412 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc\") pod \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " Oct 31 00:42:46.042447 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042433 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle\") pod \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042467 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042501 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042526 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042553 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042583 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042627 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042650 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042682 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042732 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042753 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042768 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.042819 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042808 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config" pod "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" (UID: "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042826 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042839 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n\" (UniqueName: \"kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042962 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") pod \"24e241e5-9087-4179-97c9-4707d39d7718\" (UID: \"24e241e5-9087-4179-97c9-4707d39d7718\") " Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.042997 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043057 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token\") pod \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\" (UID: \"ae4c42e2-37c4-4720-b729-4d45fcd67e1e\") " Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043092 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http\") pod \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043203 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043250 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:42:46.043612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043226 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd\" (UniqueName: \"kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd\") pod \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\" (UID: \"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1\") " Oct 31 00:42:46.044237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043693 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.044237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043715 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.044237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043729 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.044237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043743 5073 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-varlogpods\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.044237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043756 5073 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-trusted-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.044237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.043761 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.045409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.045386 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:46.045530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.045480 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc" pod "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" (UID: "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.045738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.045707 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.045738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.045710 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle" pod "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" (UID: "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.046153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.045768 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:42:46.046153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.046079 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.046327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.046286 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "test-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.046626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.046584 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd" pod "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" (UID: "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1"). InnerVolumeSpecName "kube-api-access-vnncd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.046915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.046716 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.047329 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.047292 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.048145 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.048109 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "kube-api-access-nlp8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.048585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.048556 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.048745 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.048722 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.048995 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.048980 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817" pod "24e241e5-9087-4179-97c9-4707d39d7718" (UID: "24e241e5-9087-4179-97c9-4707d39d7718"). InnerVolumeSpecName "pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:42:46.049247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.049225 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http" pod "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" (UID: "fca95d77-aeb8-48a0-a16a-3decd9bdbbe1"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.049332 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.049282 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "kube-api-access-l2nz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.051154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.051132 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint" pod "ae4c42e2-37c4-4720-b729-4d45fcd67e1e" (UID: "ae4c42e2-37c4-4720-b729-4d45fcd67e1e"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.144892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144869 5073 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-entrypoint\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.144892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144890 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144900 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144909 5073 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-datadir\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144918 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144927 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144936 5073 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-metrics\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144944 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlp8n\" (UniqueName: \"kubernetes.io/projected/24e241e5-9087-4179-97c9-4707d39d7718-kube-api-access-nlp8n\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144966 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144975 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2nz2\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-kube-api-access-l2nz2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144984 5073 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-sa-token\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.144992 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.145001 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnncd\" (UniqueName: \"kubernetes.io/projected/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-kube-api-access-vnncd\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.145009 5073 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-tmp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.145017 5073 reconciler_common.go:299] "Volume detached for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/24e241e5-9087-4179-97c9-4707d39d7718-test-logs-loki-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.145025 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.145034 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1-data-science-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.145063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.145042 5073 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/ae4c42e2-37c4-4720-b729-4d45fcd67e1e-config-openshift-service-cacrt\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.156011 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.155994 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:42:46.245969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.245942 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-1b10f005-2f1b-4ca2-a9ed-1eecfa0424a9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05d6f4fcbb1b8e817\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-f3e313e21636c4c870fea1828de39353c6c1dea16aca7208a4e2c407b638a05a\"" Oct 31 00:42:46.254165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.254139 5073 generic.go:356] "Generic (PLEG): container finished" podID="ae4c42e2-37c4-4720-b729-4d45fcd67e1e" containerID="d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852" exitCode=0 Oct 31 00:42:46.254295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.254184 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" event={"ID":"ae4c42e2-37c4-4720-b729-4d45fcd67e1e","Type":"ContainerDied","Data":"d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852"} Oct 31 00:42:46.254295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.254202 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" Oct 31 00:42:46.254295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.254233 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-9pnbq" event={"ID":"ae4c42e2-37c4-4720-b729-4d45fcd67e1e","Type":"ContainerDied","Data":"fcea90a13b09a07209336af11e6c2e9e9d474dcbf2566eec505c76d728f933de"} Oct 31 00:42:46.254295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.254268 5073 scope.go:122] "RemoveContainer" containerID="d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852" Oct 31 00:42:46.255662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.255641 5073 generic.go:356] "Generic (PLEG): container finished" podID="fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" containerID="8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb" exitCode=0 Oct 31 00:42:46.255770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.255680 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" Oct 31 00:42:46.255770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.255732 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" event={"ID":"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1","Type":"ContainerDied","Data":"8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb"} Oct 31 00:42:46.255931 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.255767 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp" event={"ID":"fca95d77-aeb8-48a0-a16a-3decd9bdbbe1","Type":"ContainerDied","Data":"acec0fbde816f0615925f02b93ce51ea3e476d93e02f8ab870327de51fef8fc7"} Oct 31 00:42:46.257402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.257384 5073 generic.go:356] "Generic (PLEG): container finished" podID="24e241e5-9087-4179-97c9-4707d39d7718" containerID="bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41" exitCode=0 Oct 31 00:42:46.257483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.257447 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 31 00:42:46.257483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.257472 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"24e241e5-9087-4179-97c9-4707d39d7718","Type":"ContainerDied","Data":"bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41"} Oct 31 00:42:46.257600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.257506 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"24e241e5-9087-4179-97c9-4707d39d7718","Type":"ContainerDied","Data":"1e856be304f7681623e2d1e4386e5fc4d35845b506978634bf42c1182a04e984"} Oct 31 00:42:46.259631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.259605 5073 generic.go:356] "Generic (PLEG): container finished" podID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerID="8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441" exitCode=0 Oct 31 00:42:46.259746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.259635 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" event={"ID":"41c54ace-b3d3-4297-922d-9082e4584ae6","Type":"ContainerDied","Data":"8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441"} Oct 31 00:42:46.266025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.266010 5073 scope.go:122] "RemoveContainer" containerID="d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852" Oct 31 00:42:46.266298 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:46.266277 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852\": container with ID starting with d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852 not found: ID does not exist" containerID="d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852" Oct 31 00:42:46.266369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.266308 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852"} err="failed to get container status \"d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852\": rpc error: code = NotFound desc = could not find container \"d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852\": container with ID starting with d2f9ae7b78f1c6fcfefdc5ee04e163c111eb7a15c1e87ba83d13e16e2f34f852 not found: ID does not exist" Oct 31 00:42:46.266369 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.266331 5073 scope.go:122] "RemoveContainer" containerID="8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb" Oct 31 00:42:46.274321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.274301 5073 scope.go:122] "RemoveContainer" containerID="8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb" Oct 31 00:42:46.274569 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:46.274551 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb\": container with ID starting with 8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb not found: ID does not exist" containerID="8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb" Oct 31 00:42:46.274642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.274574 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb"} err="failed to get container status \"8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb\": rpc error: code = NotFound desc = could not find container \"8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb\": container with ID starting with 8ae8b5ff7a71168617d12865076cbac832686aa64dc33d033e8a9fa878afa3bb not found: ID does not exist" Oct 31 00:42:46.274642 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.274588 5073 scope.go:122] "RemoveContainer" containerID="bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41" Oct 31 00:42:46.282637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.282602 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp"] Oct 31 00:42:46.283752 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.283734 5073 scope.go:122] "RemoveContainer" containerID="bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41" Oct 31 00:42:46.284041 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:46.284018 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41\": container with ID starting with bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41 not found: ID does not exist" containerID="bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41" Oct 31 00:42:46.284093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.284051 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41"} err="failed to get container status \"bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41\": rpc error: code = NotFound desc = could not find container \"bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41\": container with ID starting with bfad4c5b89157d4233b676ff2a4a9a28bd5b72e8b57ed729361dc6f5d6367e41 not found: ID does not exist" Oct 31 00:42:46.284133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.284123 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-69hxp"] Oct 31 00:42:46.296758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.296738 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 31 00:42:46.301849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.301831 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 31 00:42:46.315309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.315281 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-9pnbq"] Oct 31 00:42:46.320068 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.320023 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-9pnbq"] Oct 31 00:42:46.848581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.848546 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:42:46.952276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952251 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952303 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952349 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952413 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952370 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952410 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952477 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952503 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952580 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http\") pod \"41c54ace-b3d3-4297-922d-9082e4584ae6\" (UID: \"41c54ace-b3d3-4297-922d-9082e4584ae6\") " Oct 31 00:42:46.952865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.952657 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.954199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.954171 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.955221 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.955137 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.955527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.955498 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.956880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.956858 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "kube-api-access-jmmdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.957039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.956954 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.958708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.958688 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.959207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.959187 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.962917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:46.962351 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants" pod "41c54ace-b3d3-4297-922d-9082e4584ae6" (UID: "41c54ace-b3d3-4297-922d-9082e4584ae6"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:47.054714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054684 5073 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-lokistack-gateway\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.054882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054718 5073 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-tls-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.054882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054734 5073 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/41c54ace-b3d3-4297-922d-9082e4584ae6-rbac\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.054882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054750 5073 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-tenants\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.054882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054764 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.054882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054809 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmdn\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-kube-api-access-jmmdn\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.054882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.054826 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/41c54ace-b3d3-4297-922d-9082e4584ae6-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.074015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.073992 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:42:47.155619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.155590 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config\") pod \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " Oct 31 00:42:47.155777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.155666 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle\") pod \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " Oct 31 00:42:47.155777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.155735 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc\") pod \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " Oct 31 00:42:47.155899 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.155850 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http\") pod \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " Oct 31 00:42:47.155899 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.155881 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx\" (UniqueName: \"kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx\") pod \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\" (UID: \"a5cc166c-dd4f-42a0-b3cc-01296493c45f\") " Oct 31 00:42:47.156072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.156050 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config" pod "a5cc166c-dd4f-42a0-b3cc-01296493c45f" (UID: "a5cc166c-dd4f-42a0-b3cc-01296493c45f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:47.156146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.156089 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle" pod "a5cc166c-dd4f-42a0-b3cc-01296493c45f" (UID: "a5cc166c-dd4f-42a0-b3cc-01296493c45f"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:47.156450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.156424 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.156720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.156456 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.158194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.158166 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http" pod "a5cc166c-dd4f-42a0-b3cc-01296493c45f" (UID: "a5cc166c-dd4f-42a0-b3cc-01296493c45f"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:47.158297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.158232 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx" pod "a5cc166c-dd4f-42a0-b3cc-01296493c45f" (UID: "a5cc166c-dd4f-42a0-b3cc-01296493c45f"). InnerVolumeSpecName "kube-api-access-mkbqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:47.158567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.158549 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc" pod "a5cc166c-dd4f-42a0-b3cc-01296493c45f" (UID: "a5cc166c-dd4f-42a0-b3cc-01296493c45f"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:47.257808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.257758 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.257808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.257805 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/a5cc166c-dd4f-42a0-b3cc-01296493c45f-data-science-lokistack-distributor-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.258003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.257817 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbqx\" (UniqueName: \"kubernetes.io/projected/a5cc166c-dd4f-42a0-b3cc-01296493c45f-kube-api-access-mkbqx\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.266958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.266933 5073 generic.go:356] "Generic (PLEG): container finished" podID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerID="4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a" exitCode=0 Oct 31 00:42:47.267095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.266980 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" Oct 31 00:42:47.267095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.267031 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" event={"ID":"41c54ace-b3d3-4297-922d-9082e4584ae6","Type":"ContainerDied","Data":"4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a"} Oct 31 00:42:47.267095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.267069 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg" event={"ID":"41c54ace-b3d3-4297-922d-9082e4584ae6","Type":"ContainerDied","Data":"b9e85e3aee0b9487545db80cb4d9e3ff440eb5a4dbcdb0f6900b8ffa6430cc6b"} Oct 31 00:42:47.267095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.267090 5073 scope.go:122] "RemoveContainer" containerID="8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441" Oct 31 00:42:47.268425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.268396 5073 generic.go:356] "Generic (PLEG): container finished" podID="a5cc166c-dd4f-42a0-b3cc-01296493c45f" containerID="9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99" exitCode=0 Oct 31 00:42:47.268550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.268420 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" event={"ID":"a5cc166c-dd4f-42a0-b3cc-01296493c45f","Type":"ContainerDied","Data":"9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99"} Oct 31 00:42:47.268550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.268449 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" Oct 31 00:42:47.268640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.268453 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59" event={"ID":"a5cc166c-dd4f-42a0-b3cc-01296493c45f","Type":"ContainerDied","Data":"4874f228470c7a203b6e0e6abb91102fc31d21788404c6a335a4f3d4db58739e"} Oct 31 00:42:47.276470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.276452 5073 scope.go:122] "RemoveContainer" containerID="4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a" Oct 31 00:42:47.284939 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.284921 5073 scope.go:122] "RemoveContainer" containerID="8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441" Oct 31 00:42:47.285192 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:47.285175 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441\": container with ID starting with 8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441 not found: ID does not exist" containerID="8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441" Oct 31 00:42:47.285241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.285200 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441"} err="failed to get container status \"8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441\": rpc error: code = NotFound desc = could not find container \"8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441\": container with ID starting with 8c8aaf6b7e93374e7856a6793b80e7838df915eee3446d5c528d2b3091307441 not found: ID does not exist" Oct 31 00:42:47.285241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.285217 5073 scope.go:122] "RemoveContainer" containerID="4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a" Oct 31 00:42:47.285456 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:47.285436 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a\": container with ID starting with 4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a not found: ID does not exist" containerID="4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a" Oct 31 00:42:47.285528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.285465 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a"} err="failed to get container status \"4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a\": rpc error: code = NotFound desc = could not find container \"4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a\": container with ID starting with 4b3174ca8c7828789aa48e81b7f8605b9ba61cabcfe404bc0175d6d76255dd6a not found: ID does not exist" Oct 31 00:42:47.285528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.285487 5073 scope.go:122] "RemoveContainer" containerID="9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99" Oct 31 00:42:47.293491 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.293470 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg"] Oct 31 00:42:47.293567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.293493 5073 scope.go:122] "RemoveContainer" containerID="9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99" Oct 31 00:42:47.293974 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:47.293953 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99\": container with ID starting with 9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99 not found: ID does not exist" containerID="9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99" Oct 31 00:42:47.294087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.293989 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99"} err="failed to get container status \"9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99\": rpc error: code = NotFound desc = could not find container \"9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99\": container with ID starting with 9d94c95ea9eb0630ff4d078e5ccc2f4fee7e126e837b2e719329956c0f821a99 not found: ID does not exist" Oct 31 00:42:47.296271 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.296248 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-cpnvg"] Oct 31 00:42:47.306614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.306592 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59"] Oct 31 00:42:47.310755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:47.310733 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-b854ccfb4-gjn59"] Oct 31 00:42:48.058623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:48.058591 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e241e5-9087-4179-97c9-4707d39d7718" path="/var/lib/kubelet/pods/24e241e5-9087-4179-97c9-4707d39d7718/volumes" Oct 31 00:42:48.059016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:48.058973 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" path="/var/lib/kubelet/pods/41c54ace-b3d3-4297-922d-9082e4584ae6/volumes" Oct 31 00:42:48.059354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:48.059341 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cc166c-dd4f-42a0-b3cc-01296493c45f" path="/var/lib/kubelet/pods/a5cc166c-dd4f-42a0-b3cc-01296493c45f/volumes" Oct 31 00:42:48.060037 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:48.060023 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4c42e2-37c4-4720-b729-4d45fcd67e1e" path="/var/lib/kubelet/pods/ae4c42e2-37c4-4720-b729-4d45fcd67e1e/volumes" Oct 31 00:42:48.060368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:48.060356 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" path="/var/lib/kubelet/pods/fca95d77-aeb8-48a0-a16a-3decd9bdbbe1/volumes" Oct 31 00:42:50.948958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:50.948919 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:53.013836 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.013800 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:42:53.014301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.013972 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" containerID="cri-o://dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8" gracePeriod=600 Oct 31 00:42:53.041472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.041433 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:42:53.041655 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.041628 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="ebfefa87-b20a-4dc0-a543-a647be527cd5" containerName="loki-compactor" containerID="cri-o://df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b" gracePeriod=30 Oct 31 00:42:53.044615 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.044585 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n"] Oct 31 00:42:53.045025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045000 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" containerName="loki-query-frontend" Oct 31 00:42:53.045025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045024 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" containerName="loki-query-frontend" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045037 5073 cpu_manager.go:414] "containerMap: removing container" podUID="24e241e5-9087-4179-97c9-4707d39d7718" containerName="loki-index-gateway" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045043 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e241e5-9087-4179-97c9-4707d39d7718" containerName="loki-index-gateway" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045051 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ae4c42e2-37c4-4720-b729-4d45fcd67e1e" containerName="collector" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045056 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4c42e2-37c4-4720-b729-4d45fcd67e1e" containerName="collector" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045069 5073 cpu_manager.go:414] "containerMap: removing container" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="opa" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045074 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="opa" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045100 5073 cpu_manager.go:414] "containerMap: removing container" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="gateway" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045104 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="gateway" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045114 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a5cc166c-dd4f-42a0-b3cc-01296493c45f" containerName="loki-distributor" Oct 31 00:42:53.045150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045119 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cc166c-dd4f-42a0-b3cc-01296493c45f" containerName="loki-distributor" Oct 31 00:42:53.045488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045180 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cc166c-dd4f-42a0-b3cc-01296493c45f" containerName="loki-distributor" Oct 31 00:42:53.045488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045190 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca95d77-aeb8-48a0-a16a-3decd9bdbbe1" containerName="loki-query-frontend" Oct 31 00:42:53.045488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045196 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e241e5-9087-4179-97c9-4707d39d7718" containerName="loki-index-gateway" Oct 31 00:42:53.045488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045201 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4c42e2-37c4-4720-b729-4d45fcd67e1e" containerName="collector" Oct 31 00:42:53.045488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045211 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="opa" Oct 31 00:42:53.045488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.045218 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c54ace-b3d3-4297-922d-9082e4584ae6" containerName="gateway" Oct 31 00:42:53.049912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.049892 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.058287 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.058266 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 31 00:42:53.058446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.058311 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 31 00:42:53.058580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.058551 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 31 00:42:53.060213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.060195 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n"] Oct 31 00:42:53.205086 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.205051 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.205275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.205105 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.205275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.205199 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.205275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.205246 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nfj\" (UniqueName: \"kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.205275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.205268 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.205415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.205324 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.305924 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.305969 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.305998 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.306021 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nfj\" (UniqueName: \"kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.306038 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.306076 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.306969 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.306945 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.307102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.307080 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.308746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.308721 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.309168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.309149 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.309308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.309278 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.318953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.318931 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nfj\" (UniqueName: \"kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.361953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.361922 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:53.642306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.642277 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n"] Oct 31 00:42:53.644917 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:42:53.644882 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0237536f_fffe_4152_91f7_cf98e84d3c77.slice/crio-7a378c74296a89313a514b89260e64563ad77574beac8a6de36b5846d03bfee3 WatchSource:0}: Error finding container 7a378c74296a89313a514b89260e64563ad77574beac8a6de36b5846d03bfee3: Status 404 returned error can't find the container with id 7a378c74296a89313a514b89260e64563ad77574beac8a6de36b5846d03bfee3 Oct 31 00:42:53.746717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.746661 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:42:53.750355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.750337 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.755050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.755023 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-grpc\"" Oct 31 00:42:53.755050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.755047 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-http\"" Oct 31 00:42:53.760222 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.760189 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:42:53.914824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.913149 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.914824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.913210 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.914824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.913235 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.914824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.913267 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.914824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.913292 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.914824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.913340 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nph2\" (UniqueName: \"kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.995727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:53.995701 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.013864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.013838 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nph2\" (UniqueName: \"kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.014245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.013895 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.014245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.013922 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.014245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.013945 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.014245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.013982 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.014245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.014007 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.014736 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.014687 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.015096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.015072 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.017167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.017138 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.017950 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.017246 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.018454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.018431 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.028424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.028397 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nph2\" (UniqueName: \"kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:54.040067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.040044 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.114803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.114761 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.114944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.114844 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.114944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.114881 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.114944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.114915 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.115148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.114944 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.115148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.114977 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8\" (UniqueName: \"kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.115148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115069 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.115148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115142 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.115379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115164 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:54.115379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115193 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.115379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115230 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.115379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115297 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.115379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115324 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.115647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115420 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.115647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115452 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle\") pod \"e0738c47-24ab-414d-99b7-181edf6477ff\" (UID: \"e0738c47-24ab-414d-99b7-181edf6477ff\") " Oct 31 00:42:54.115647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115500 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r\" (UniqueName: \"kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r\") pod \"ebfefa87-b20a-4dc0-a543-a647be527cd5\" (UID: \"ebfefa87-b20a-4dc0-a543-a647be527cd5\") " Oct 31 00:42:54.115891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.115870 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.116301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.116022 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:54.116301 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.116257 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:54.116865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.116633 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:54.119748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.119724 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:42:54.119975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.119766 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "pvc-81c40fd7-f377-4fca-a7c3-6d14328be212". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:42:54.121760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.121714 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.121973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.121948 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.122274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.122229 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.122429 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.122393 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "kube-api-access-22w8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:54.122628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.122591 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.126837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.122673 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.126946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.122881 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "pvc-e618b476-df27-42e1-92f9-d875e765a448". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:42:54.126946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.123189 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8" pod "e0738c47-24ab-414d-99b7-181edf6477ff" (UID: "e0738c47-24ab-414d-99b7-181edf6477ff"). InnerVolumeSpecName "kube-api-access-gnbl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:54.126946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.125247 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc" pod "ebfefa87-b20a-4dc0-a543-a647be527cd5" (UID: "ebfefa87-b20a-4dc0-a543-a647be527cd5"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.216445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216379 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216405 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216417 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22w8r\" (UniqueName: \"kubernetes.io/projected/ebfefa87-b20a-4dc0-a543-a647be527cd5-kube-api-access-22w8r\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216427 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216436 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-data-science-usage-logs-lokistack-ingester-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216445 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216446 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216454 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216464 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbl8\" (UniqueName: \"kubernetes.io/projected/e0738c47-24ab-414d-99b7-181edf6477ff-kube-api-access-gnbl8\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216490 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216503 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216512 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0738c47-24ab-414d-99b7-181edf6477ff-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216521 5073 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/e0738c47-24ab-414d-99b7-181edf6477ff-test-usage-logs-loki-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216534 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:42:54.216774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.216542 5073 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/ebfefa87-b20a-4dc0-a543-a647be527cd5-test-usage-logs-loki-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.227741 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.227722 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:42:54.227991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.227974 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-81c40fd7-f377-4fca-a7c3-6d14328be212" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:42:54.230519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.230504 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e618b476-df27-42e1-92f9-d875e765a448" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:42:54.309794 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.309752 5073 generic.go:356] "Generic (PLEG): container finished" podID="e0738c47-24ab-414d-99b7-181edf6477ff" containerID="dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8" exitCode=0 Oct 31 00:42:54.309897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.309863 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.309948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.309888 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"e0738c47-24ab-414d-99b7-181edf6477ff","Type":"ContainerDied","Data":"dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8"} Oct 31 00:42:54.309948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.309935 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"e0738c47-24ab-414d-99b7-181edf6477ff","Type":"ContainerDied","Data":"524850d0d79ae9b0cb526eff7f04e417e12ba2f63c0b23f30b5885a2a6e8db08"} Oct 31 00:42:54.310052 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.309951 5073 scope.go:122] "RemoveContainer" containerID="dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8" Oct 31 00:42:54.311569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.311545 5073 generic.go:356] "Generic (PLEG): container finished" podID="ebfefa87-b20a-4dc0-a543-a647be527cd5" containerID="df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b" exitCode=0 Oct 31 00:42:54.311817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.311613 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.311919 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.311629 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"ebfefa87-b20a-4dc0-a543-a647be527cd5","Type":"ContainerDied","Data":"df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b"} Oct 31 00:42:54.311977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.311931 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"ebfefa87-b20a-4dc0-a543-a647be527cd5","Type":"ContainerDied","Data":"c91a7fe2f906f44e1d9ac9c62ccb9f21220ce5f7473c7ab4df0f7608afdf39fa"} Oct 31 00:42:54.315001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.314974 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" event={"ID":"0237536f-fffe-4152-91f7-cf98e84d3c77","Type":"ContainerStarted","Data":"f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98"} Oct 31 00:42:54.315106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.315017 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" event={"ID":"0237536f-fffe-4152-91f7-cf98e84d3c77","Type":"ContainerStarted","Data":"7a378c74296a89313a514b89260e64563ad77574beac8a6de36b5846d03bfee3"} Oct 31 00:42:54.315166 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.315111 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:42:54.317146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.317124 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-70efa0f79eac0e40982684e61df4f3833412c2cd0157e43aa1a3aa8d5e4dbb9e\"" Oct 31 00:42:54.317206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.317152 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-e6fd39fc46b3d77f941fdbebf9769b8424ddd772f87e48a52f180c692830420f\"" Oct 31 00:42:54.317206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.317168 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-0e3ae4b7a4a45b7fac2e73d30f486b2b5989e25d15f0b8f632bc09c0fb516836\"" Oct 31 00:42:54.338376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.338336 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" podStartSLOduration=1.338324728 podStartE2EDuration="1.338324728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:54.33679889 +0000 UTC m=+2658.397573585" watchObservedRunningTime="2026-10-31 00:42:54.338324728 +0000 UTC m=+2658.399099417" Oct 31 00:42:54.341717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.341703 5073 scope.go:122] "RemoveContainer" containerID="dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8" Oct 31 00:42:54.342008 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:54.341986 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8\": container with ID starting with dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8 not found: ID does not exist" containerID="dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8" Oct 31 00:42:54.342062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.342015 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8"} err="failed to get container status \"dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8\": rpc error: code = NotFound desc = could not find container \"dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8\": container with ID starting with dd2bcd47d053a1c56d736eb2a2fdf93f2433e6cf7042fd49e11a1e9eaa438ac8 not found: ID does not exist" Oct 31 00:42:54.342062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.342030 5073 scope.go:122] "RemoveContainer" containerID="df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b" Oct 31 00:42:54.351046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.351027 5073 scope.go:122] "RemoveContainer" containerID="df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b" Oct 31 00:42:54.351328 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:42:54.351308 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b\": container with ID starting with df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b not found: ID does not exist" containerID="df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b" Oct 31 00:42:54.351420 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.351331 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b"} err="failed to get container status \"df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b\": rpc error: code = NotFound desc = could not find container \"df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b\": container with ID starting with df3d0e296ab0797bed52a7131ff2e6837f3267e0023efc7cf96f0d96f2c0a25b not found: ID does not exist" Oct 31 00:42:54.355689 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.355667 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:42:54.359977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.359958 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:42:54.370774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.370755 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:42:54.380041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.380022 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:42:54.386128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386108 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:42:54.386463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386450 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ebfefa87-b20a-4dc0-a543-a647be527cd5" containerName="loki-compactor" Oct 31 00:42:54.386507 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386464 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfefa87-b20a-4dc0-a543-a647be527cd5" containerName="loki-compactor" Oct 31 00:42:54.386547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386507 5073 cpu_manager.go:414] "containerMap: removing container" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" Oct 31 00:42:54.386547 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386513 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" Oct 31 00:42:54.386608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386566 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfefa87-b20a-4dc0-a543-a647be527cd5" containerName="loki-compactor" Oct 31 00:42:54.386608 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.386578 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" containerName="loki-ingester" Oct 31 00:42:54.395704 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.395689 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.401243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.401218 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:42:54.445808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.445761 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-grpc\"" Oct 31 00:42:54.445927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.445876 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-http\"" Oct 31 00:42:54.518562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.518538 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.518698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.518602 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.518698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.518663 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.518937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.518711 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4b8\" (UniqueName: \"kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.518937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.518770 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.518937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.518823 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.619343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.619311 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.619454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.619347 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.619454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.619379 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.619454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.619428 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.619454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.619444 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.619662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.619461 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4b8\" (UniqueName: \"kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.620217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.620189 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.620334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.620277 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.622061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.622042 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.622572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.622549 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.622683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.622641 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:54.632143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:54.632106 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4b8\" (UniqueName: \"kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:56.058594 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:56.058553 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0738c47-24ab-414d-99b7-181edf6477ff" path="/var/lib/kubelet/pods/e0738c47-24ab-414d-99b7-181edf6477ff/volumes" Oct 31 00:42:56.059172 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:56.059151 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebfefa87-b20a-4dc0-a543-a647be527cd5" path="/var/lib/kubelet/pods/ebfefa87-b20a-4dc0-a543-a647be527cd5/volumes" Oct 31 00:42:59.767348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.767315 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.767742 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.767360 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.774355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.774329 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.774474 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.774407 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.868700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.868676 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.868868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.868709 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.868868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.868832 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.868868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.868860 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.872453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.872435 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") DevicePath \"csi-e6fd39fc46b3d77f941fdbebf9769b8424ddd772f87e48a52f180c692830420f\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.872521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.872503 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") DevicePath \"csi-70efa0f79eac0e40982684e61df4f3833412c2cd0157e43aa1a3aa8d5e4dbb9e\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.966982 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.966947 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fdf3d44672bff7110481cf42cbe6e99aa39e5c8305a671c1179f840d2ab17bcc/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.968054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.968024 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7c9d44c2ac5b185f6db2eab8b6cc11f2151bd1a519bd148465043863d898a7a3/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.981391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.981369 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:42:59.987069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:42:59.987048 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:43:00.109007 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:00.108951 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:43:00.388980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:00.388811 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:43:00.391603 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:43:00.391578 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86c84e14_f511_4175_ab01_5ca13b120d14.slice/crio-9377766727bf126a5cda80e708156623bfd03f3f2e8d958945b01be904e355b2 WatchSource:0}: Error finding container 9377766727bf126a5cda80e708156623bfd03f3f2e8d958945b01be904e355b2: Status 404 returned error can't find the container with id 9377766727bf126a5cda80e708156623bfd03f3f2e8d958945b01be904e355b2 Oct 31 00:43:00.949108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:00.949069 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:01.352964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:01.352933 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"86c84e14-f511-4175-ab01-5ca13b120d14","Type":"ContainerStarted","Data":"dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f"} Oct 31 00:43:01.353165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:01.352971 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"86c84e14-f511-4175-ab01-5ca13b120d14","Type":"ContainerStarted","Data":"9377766727bf126a5cda80e708156623bfd03f3f2e8d958945b01be904e355b2"} Oct 31 00:43:01.353165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:01.353000 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:43:01.375393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:01.375346 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podStartSLOduration=7.375332591 podStartE2EDuration="7.375332591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:01.374064297 +0000 UTC m=+2665.434838989" watchObservedRunningTime="2026-10-31 00:43:01.375332591 +0000 UTC m=+2665.436107286" Oct 31 00:43:04.508967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.508933 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.515531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.515501 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.609684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.609656 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.609834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.609752 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.613443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.613422 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") DevicePath \"csi-d4654be4ad7375db49ffa2409572339f1bb3740d7436b5c76259d4d03f9052a3\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.704641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.704609 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8fd5a5ac985486c6416e9f57ebb5ef27608808414a3869903174362f39d872c7/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.719516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.719490 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:04.923613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:04.923547 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:05.230093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:05.230064 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:43:05.232502 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:43:05.232464 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f0c53e_10cc_41ec_ba07_1ead5f1b08dc.slice/crio-8ae582f721c149f80b0ec33ed88c6d5bf4062a45760860c000b2905ad3306439 WatchSource:0}: Error finding container 8ae582f721c149f80b0ec33ed88c6d5bf4062a45760860c000b2905ad3306439: Status 404 returned error can't find the container with id 8ae582f721c149f80b0ec33ed88c6d5bf4062a45760860c000b2905ad3306439 Oct 31 00:43:05.374562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:05.374525 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc","Type":"ContainerStarted","Data":"740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf"} Oct 31 00:43:05.374708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:05.374574 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc","Type":"ContainerStarted","Data":"8ae582f721c149f80b0ec33ed88c6d5bf4062a45760860c000b2905ad3306439"} Oct 31 00:43:05.374708 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:05.374612 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:05.396410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:05.396360 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podStartSLOduration=12.396348419 podStartE2EDuration="12.396348419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:05.395224539 +0000 UTC m=+2669.455999287" watchObservedRunningTime="2026-10-31 00:43:05.396348419 +0000 UTC m=+2669.457123113" Oct 31 00:43:10.949238 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:10.949187 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:10.949657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:10.949300 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" Oct 31 00:43:13.085165 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:13.085128 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:43:15.327911 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:15.327881 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:43:20.949391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:20.949351 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:22.363315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:22.363274 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:26.384858 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:26.384830 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:30.949061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:30.949023 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:32.361845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:32.361809 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:36.352350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:36.352249 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:43:36.359748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:36.359729 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:43:36.360768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:36.360752 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:43:36.368188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:36.368170 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:43:37.213960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:37.213928 5073 scope.go:122] "RemoveContainer" containerID="28c0fc704324759991f634859f98138ee1cb53849ce788d2c59c83d932ddd8e4" Oct 31 00:43:37.222617 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:37.222597 5073 scope.go:122] "RemoveContainer" containerID="4fe063f34a499c6a1851c9dcc0c33d35c459a3edf226ac1b68196ee028c5aa79" Oct 31 00:43:40.948853 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:40.948813 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:42.361678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:42.361646 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:43:43.085307 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:43.085265 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:43:50.948760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:50.948720 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:56.750678 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.750642 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:43:56.751657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.751620 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podUID="65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" containerName="loki-index-gateway" containerID="cri-o://740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf" gracePeriod=30 Oct 31 00:43:56.855467 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.855419 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:43:56.855703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.855642 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" containerName="loki-ingester" containerID="cri-o://dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f" gracePeriod=600 Oct 31 00:43:56.869173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.869130 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n"] Oct 31 00:43:56.869409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.869360 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" podUID="0237536f-fffe-4152-91f7-cf98e84d3c77" containerName="loki-querier" containerID="cri-o://f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98" gracePeriod=30 Oct 31 00:43:56.872294 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.871392 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz"] Oct 31 00:43:56.872294 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.871565 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="opa" containerID="cri-o://40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03" gracePeriod=30 Oct 31 00:43:56.872294 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:56.871558 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="gateway" containerID="cri-o://fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161" gracePeriod=30 Oct 31 00:43:57.033651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.033628 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:57.129896 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.129869 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130000 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130032 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130184 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130073 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2\" (UniqueName: \"kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130184 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130156 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130198 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130231 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle\") pod \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\" (UID: \"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc\") " Oct 31 00:43:57.130882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130769 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.130882 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.130828 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.132767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.132744 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.132909 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.132888 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.133260 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.133181 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.133535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.133510 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "kube-api-access-8nph2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:57.133535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.133514 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42" pod "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" (UID: "65f0c53e-10cc-41ec-ba07-1ead5f1b08dc"). InnerVolumeSpecName "pvc-d40c9d92-b1bc-4286-b370-a2b061c99081". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:43:57.231641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231613 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.231641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231639 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.231852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231654 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-data-science-usage-logs-lokistack-index-gateway-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.231852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231688 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:43:57.231852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231704 5073 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-test-loki-lifecycle-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.231852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231718 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nph2\" (UniqueName: \"kubernetes.io/projected/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-kube-api-access-8nph2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.231852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.231731 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.241067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.241051 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-d40c9d92-b1bc-4286-b370-a2b061c99081" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:43:57.332360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.332327 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-d4654be4ad7375db49ffa2409572339f1bb3740d7436b5c76259d4d03f9052a3\"" Oct 31 00:43:57.425192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.425170 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:43:57.533565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.533496 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj\" (UniqueName: \"kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj\") pod \"0237536f-fffe-4152-91f7-cf98e84d3c77\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " Oct 31 00:43:57.533565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.533551 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle\") pod \"0237536f-fffe-4152-91f7-cf98e84d3c77\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " Oct 31 00:43:57.533731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.533608 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret\") pod \"0237536f-fffe-4152-91f7-cf98e84d3c77\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " Oct 31 00:43:57.533731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.533631 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config\") pod \"0237536f-fffe-4152-91f7-cf98e84d3c77\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " Oct 31 00:43:57.533731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.533654 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http\") pod \"0237536f-fffe-4152-91f7-cf98e84d3c77\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " Oct 31 00:43:57.533731 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.533722 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc\") pod \"0237536f-fffe-4152-91f7-cf98e84d3c77\" (UID: \"0237536f-fffe-4152-91f7-cf98e84d3c77\") " Oct 31 00:43:57.534051 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.534018 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle" pod "0237536f-fffe-4152-91f7-cf98e84d3c77" (UID: "0237536f-fffe-4152-91f7-cf98e84d3c77"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.534175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.534074 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config" pod "0237536f-fffe-4152-91f7-cf98e84d3c77" (UID: "0237536f-fffe-4152-91f7-cf98e84d3c77"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.535900 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.535873 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj" pod "0237536f-fffe-4152-91f7-cf98e84d3c77" (UID: "0237536f-fffe-4152-91f7-cf98e84d3c77"). InnerVolumeSpecName "kube-api-access-t8nfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:57.535990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.535946 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret" pod "0237536f-fffe-4152-91f7-cf98e84d3c77" (UID: "0237536f-fffe-4152-91f7-cf98e84d3c77"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.536055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.536033 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc" pod "0237536f-fffe-4152-91f7-cf98e84d3c77" (UID: "0237536f-fffe-4152-91f7-cf98e84d3c77"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.536097 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.536068 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http" pod "0237536f-fffe-4152-91f7-cf98e84d3c77" (UID: "0237536f-fffe-4152-91f7-cf98e84d3c77"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.625220 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.625194 5073 generic.go:356] "Generic (PLEG): container finished" podID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerID="40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03" exitCode=0 Oct 31 00:43:57.625321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.625268 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" event={"ID":"f2740588-0d1b-47d6-a95c-1ea7b0332673","Type":"ContainerDied","Data":"40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03"} Oct 31 00:43:57.626492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.626470 5073 generic.go:356] "Generic (PLEG): container finished" podID="0237536f-fffe-4152-91f7-cf98e84d3c77" containerID="f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98" exitCode=0 Oct 31 00:43:57.626585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.626515 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" Oct 31 00:43:57.626585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.626554 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" event={"ID":"0237536f-fffe-4152-91f7-cf98e84d3c77","Type":"ContainerDied","Data":"f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98"} Oct 31 00:43:57.626681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.626592 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n" event={"ID":"0237536f-fffe-4152-91f7-cf98e84d3c77","Type":"ContainerDied","Data":"7a378c74296a89313a514b89260e64563ad77574beac8a6de36b5846d03bfee3"} Oct 31 00:43:57.626681 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.626611 5073 scope.go:122] "RemoveContainer" containerID="f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98" Oct 31 00:43:57.627902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.627882 5073 generic.go:356] "Generic (PLEG): container finished" podID="65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" containerID="740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf" exitCode=0 Oct 31 00:43:57.627999 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.627922 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:43:57.627999 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.627958 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc","Type":"ContainerDied","Data":"740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf"} Oct 31 00:43:57.627999 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.627989 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"65f0c53e-10cc-41ec-ba07-1ead5f1b08dc","Type":"ContainerDied","Data":"8ae582f721c149f80b0ec33ed88c6d5bf4062a45760860c000b2905ad3306439"} Oct 31 00:43:57.634424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.634403 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.634515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.634426 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.634515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.634442 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8nfj\" (UniqueName: \"kubernetes.io/projected/0237536f-fffe-4152-91f7-cf98e84d3c77-kube-api-access-t8nfj\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.634515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.634457 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.634515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.634473 5073 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/0237536f-fffe-4152-91f7-cf98e84d3c77-test-loki-lifecycle-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.634515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.634487 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0237536f-fffe-4152-91f7-cf98e84d3c77-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.651884 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.651864 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n"] Oct 31 00:43:57.656549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.656528 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-pgz6n"] Oct 31 00:43:57.667406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.667386 5073 scope.go:122] "RemoveContainer" containerID="f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98" Oct 31 00:43:57.667687 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:57.667668 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98\": container with ID starting with f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98 not found: ID does not exist" containerID="f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98" Oct 31 00:43:57.667755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.667693 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98"} err="failed to get container status \"f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98\": rpc error: code = NotFound desc = could not find container \"f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98\": container with ID starting with f9a95c711e61287ca0d6b9e444394fe2a8d100d777b030748c1566837f609c98 not found: ID does not exist" Oct 31 00:43:57.667755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.667708 5073 scope.go:122] "RemoveContainer" containerID="740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf" Oct 31 00:43:57.669901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.669879 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:43:57.674815 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.674770 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:43:57.678025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.678010 5073 scope.go:122] "RemoveContainer" containerID="740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf" Oct 31 00:43:57.678319 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:57.678302 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf\": container with ID starting with 740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf not found: ID does not exist" containerID="740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf" Oct 31 00:43:57.678375 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.678325 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf"} err="failed to get container status \"740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf\": rpc error: code = NotFound desc = could not find container \"740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf\": container with ID starting with 740202e016dd69b14e9d06d8ed5a4822ac6dbf5b1a8873b152a7f62476f408bf not found: ID does not exist" Oct 31 00:43:57.821632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.821610 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:43:57.835997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.835978 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836055 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836093 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836115 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836135 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836155 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836196 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8\" (UniqueName: \"kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836236 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config\") pod \"86c84e14-f511-4175-ab01-5ca13b120d14\" (UID: \"86c84e14-f511-4175-ab01-5ca13b120d14\") " Oct 31 00:43:57.836575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.836552 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.837250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.837155 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.839016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.838989 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.839236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.839213 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:43:57.839236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.839223 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "pvc-e618b476-df27-42e1-92f9-d875e765a448". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:43:57.839334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.839299 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.839334 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.839314 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.839441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.839358 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8" pod "86c84e14-f511-4175-ab01-5ca13b120d14" (UID: "86c84e14-f511-4175-ab01-5ca13b120d14"). InnerVolumeSpecName "kube-api-access-ng4b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:57.937262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937238 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937265 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937276 5073 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-test-loki-lifecycle-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937287 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937297 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/86c84e14-f511-4175-ab01-5ca13b120d14-data-science-usage-logs-lokistack-ingester-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937308 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng4b8\" (UniqueName: \"kubernetes.io/projected/86c84e14-f511-4175-ab01-5ca13b120d14-kube-api-access-ng4b8\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937324 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86c84e14-f511-4175-ab01-5ca13b120d14-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.937374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.937348 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:43:57.947257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.947240 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:43:57.950988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:57.950973 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e618b476-df27-42e1-92f9-d875e765a448" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:43:58.037774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.037744 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-e6fd39fc46b3d77f941fdbebf9769b8424ddd772f87e48a52f180c692830420f\"" Oct 31 00:43:58.037774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.037772 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-70efa0f79eac0e40982684e61df4f3833412c2cd0157e43aa1a3aa8d5e4dbb9e\"" Oct 31 00:43:58.059976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.059909 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0237536f-fffe-4152-91f7-cf98e84d3c77" path="/var/lib/kubelet/pods/0237536f-fffe-4152-91f7-cf98e84d3c77/volumes" Oct 31 00:43:58.060545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.060520 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" path="/var/lib/kubelet/pods/65f0c53e-10cc-41ec-ba07-1ead5f1b08dc/volumes" Oct 31 00:43:58.108457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.108434 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:43:58.138391 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138370 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138441 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138488 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138644 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138615 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138682 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138768 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138728 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138764 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138828 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret\") pod \"f2740588-0d1b-47d6-a95c-1ea7b0332673\" (UID: \"f2740588-0d1b-47d6-a95c-1ea7b0332673\") " Oct 31 00:43:58.138870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.138833 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.139167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.139135 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.141399 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.141367 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "kube-api-access-xzrhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.141531 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.141511 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.142115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.142076 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.142279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.142118 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.145189 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.145159 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.146878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.146851 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.147061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.147036 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants" pod "f2740588-0d1b-47d6-a95c-1ea7b0332673" (UID: "f2740588-0d1b-47d6-a95c-1ea7b0332673"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.240411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240385 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.240411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240412 5073 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-lokistack-gateway\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.240584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240427 5073 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f2740588-0d1b-47d6-a95c-1ea7b0332673-rbac\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.240584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240441 5073 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-tenants\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.240584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240452 5073 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-tls-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.240584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240467 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/f2740588-0d1b-47d6-a95c-1ea7b0332673-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.240584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.240483 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzrhm\" (UniqueName: \"kubernetes.io/projected/f2740588-0d1b-47d6-a95c-1ea7b0332673-kube-api-access-xzrhm\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.634439 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.634408 5073 generic.go:356] "Generic (PLEG): container finished" podID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerID="fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161" exitCode=0 Oct 31 00:43:58.634626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.634461 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" Oct 31 00:43:58.634626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.634496 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" event={"ID":"f2740588-0d1b-47d6-a95c-1ea7b0332673","Type":"ContainerDied","Data":"fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161"} Oct 31 00:43:58.634626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.634536 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz" event={"ID":"f2740588-0d1b-47d6-a95c-1ea7b0332673","Type":"ContainerDied","Data":"b9eb93df819f01d2675d649e192aa9c3680f3b151c842874de53db0c65ae78d3"} Oct 31 00:43:58.634626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.634554 5073 scope.go:122] "RemoveContainer" containerID="40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03" Oct 31 00:43:58.636836 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.636815 5073 generic.go:356] "Generic (PLEG): container finished" podID="86c84e14-f511-4175-ab01-5ca13b120d14" containerID="dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f" exitCode=0 Oct 31 00:43:58.636967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.636892 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:43:58.637050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.636892 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"86c84e14-f511-4175-ab01-5ca13b120d14","Type":"ContainerDied","Data":"dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f"} Oct 31 00:43:58.637050 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.637001 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"86c84e14-f511-4175-ab01-5ca13b120d14","Type":"ContainerDied","Data":"9377766727bf126a5cda80e708156623bfd03f3f2e8d958945b01be904e355b2"} Oct 31 00:43:58.644735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.644605 5073 scope.go:122] "RemoveContainer" containerID="fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161" Oct 31 00:43:58.652816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.652798 5073 scope.go:122] "RemoveContainer" containerID="40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03" Oct 31 00:43:58.653098 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:58.653078 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03\": container with ID starting with 40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03 not found: ID does not exist" containerID="40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03" Oct 31 00:43:58.653156 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.653104 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03"} err="failed to get container status \"40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03\": rpc error: code = NotFound desc = could not find container \"40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03\": container with ID starting with 40caed70b1b010acf206483e073249c6662f643180e3d64ae3d9a0cc9ff9ac03 not found: ID does not exist" Oct 31 00:43:58.653156 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.653119 5073 scope.go:122] "RemoveContainer" containerID="fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161" Oct 31 00:43:58.653342 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:58.653327 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161\": container with ID starting with fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161 not found: ID does not exist" containerID="fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161" Oct 31 00:43:58.653384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.653346 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161"} err="failed to get container status \"fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161\": rpc error: code = NotFound desc = could not find container \"fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161\": container with ID starting with fdc7c777d225964bd4b3b7159b9ddc4822ff700ea1bcc5d4de11940f8a3f8161 not found: ID does not exist" Oct 31 00:43:58.653384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.653358 5073 scope.go:122] "RemoveContainer" containerID="dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f" Oct 31 00:43:58.657033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.657014 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:43:58.661595 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.661544 5073 scope.go:122] "RemoveContainer" containerID="dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f" Oct 31 00:43:58.662210 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:43:58.662079 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f\": container with ID starting with dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f not found: ID does not exist" containerID="dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f" Oct 31 00:43:58.662210 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.662127 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f"} err="failed to get container status \"dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f\": rpc error: code = NotFound desc = could not find container \"dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f\": container with ID starting with dbeaab9411d79b756bb41294884e45945a0c2ea0c138100b40fa7c59dfc07e6f not found: ID does not exist" Oct 31 00:43:58.663160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.663139 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:43:58.675254 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.675231 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz"] Oct 31 00:43:58.681025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:43:58.681004 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-4mffz"] Oct 31 00:44:00.058854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.058819 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" path="/var/lib/kubelet/pods/86c84e14-f511-4175-ab01-5ca13b120d14/volumes" Oct 31 00:44:00.059324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.059307 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" path="/var/lib/kubelet/pods/f2740588-0d1b-47d6-a95c-1ea7b0332673/volumes" Oct 31 00:44:00.769149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769120 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth"] Oct 31 00:44:00.769527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769503 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="gateway" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769527 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="gateway" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769549 5073 cpu_manager.go:414] "containerMap: removing container" podUID="65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" containerName="loki-index-gateway" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769557 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" containerName="loki-index-gateway" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769574 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="opa" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769581 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="opa" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769594 5073 cpu_manager.go:414] "containerMap: removing container" podUID="0237536f-fffe-4152-91f7-cf98e84d3c77" containerName="loki-querier" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769601 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0237536f-fffe-4152-91f7-cf98e84d3c77" containerName="loki-querier" Oct 31 00:44:00.769627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769626 5073 cpu_manager.go:414] "containerMap: removing container" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" containerName="loki-ingester" Oct 31 00:44:00.769920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769634 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" containerName="loki-ingester" Oct 31 00:44:00.769920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769769 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="gateway" Oct 31 00:44:00.769920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769801 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f0c53e-10cc-41ec-ba07-1ead5f1b08dc" containerName="loki-index-gateway" Oct 31 00:44:00.769920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769812 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2740588-0d1b-47d6-a95c-1ea7b0332673" containerName="opa" Oct 31 00:44:00.769920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769821 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="0237536f-fffe-4152-91f7-cf98e84d3c77" containerName="loki-querier" Oct 31 00:44:00.769920 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.769833 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c84e14-f511-4175-ab01-5ca13b120d14" containerName="loki-ingester" Oct 31 00:44:00.781004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.780980 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.783524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.783496 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth"] Oct 31 00:44:00.788764 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.788740 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 31 00:44:00.788924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.788808 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 31 00:44:00.791034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.791010 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 31 00:44:00.791493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.791216 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-lt9m8\"" Oct 31 00:44:00.791493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.791337 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 31 00:44:00.791493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.791429 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 31 00:44:00.860639 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.860609 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.860816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.860657 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.860887 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.860812 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.860926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.860887 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jsqw\" (UniqueName: \"kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.860975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.860935 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.861041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.861023 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.901589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.901556 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:44:00.905384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.905363 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.916572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.916526 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-http\"" Oct 31 00:44:00.916715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.916614 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-grpc\"" Oct 31 00:44:00.916917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.916896 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:44:00.949320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.949293 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:00.962115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962089 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jsqw\" (UniqueName: \"kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.962262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962130 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.962262 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962162 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.962393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962247 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.962393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962301 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.962393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962363 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.962566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962420 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.962566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962449 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.962566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962483 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.962566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962511 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.962566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962563 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsj6\" (UniqueName: \"kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:00.962868 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.962610 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.964913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.964708 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.964913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.964741 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.966091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.966064 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.966205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.966139 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.968061 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.968041 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.980185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.980132 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jsqw\" (UniqueName: \"kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-nhrth\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:00.990753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.990731 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp"] Oct 31 00:44:00.995582 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:00.995561 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.001383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.001307 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:44:01.001383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.001310 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 31 00:44:01.001383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.001345 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 31 00:44:01.001586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.001490 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-5jsm7\"" Oct 31 00:44:01.008024 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.008002 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp"] Oct 31 00:44:01.063451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063391 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.063451 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063441 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063512 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063558 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063592 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063708 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063737 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063766 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsj6\" (UniqueName: \"kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063828 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltszv\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.063894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063878 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.064325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063906 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.064325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063940 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.064325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.063987 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.064325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.064069 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.064325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.064266 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.064778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.064584 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.066623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.066598 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.066829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.066811 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.066980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.066959 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.077059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.077040 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsj6\" (UniqueName: \"kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:01.089414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.089393 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 31 00:44:01.089414 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.089402 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:44:01.096593 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.096572 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:01.165493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165467 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltszv\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165509 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165531 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165551 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165569 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165588 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165616 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.165974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.165658 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.166316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.166290 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.166463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.166442 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.166894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.166867 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.168837 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.168809 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.168982 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.168966 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.169491 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.169467 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.169536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.169525 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.179343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.179317 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltszv\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.315188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.315122 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:01.375331 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.375296 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth"] Oct 31 00:44:01.386079 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:44:01.385128 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114ccd9b_8cfc_463c_8a72_5e5b2ea61105.slice/crio-283a9f30d41af7a98916ee54c0d13d5693b37acc2cd66a8d75ab011dab059d35 WatchSource:0}: Error finding container 283a9f30d41af7a98916ee54c0d13d5693b37acc2cd66a8d75ab011dab059d35: Status 404 returned error can't find the container with id 283a9f30d41af7a98916ee54c0d13d5693b37acc2cd66a8d75ab011dab059d35 Oct 31 00:44:01.604614 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.604530 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp"] Oct 31 00:44:01.607351 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:44:01.607321 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c4f99bf_f802_4ae9_b736_f19e0f12af98.slice/crio-c7eebc005ec2b278cbab5191195bc732ee04d01f084e6ee4c0d6c03c200ac6ff WatchSource:0}: Error finding container c7eebc005ec2b278cbab5191195bc732ee04d01f084e6ee4c0d6c03c200ac6ff: Status 404 returned error can't find the container with id c7eebc005ec2b278cbab5191195bc732ee04d01f084e6ee4c0d6c03c200ac6ff Oct 31 00:44:01.659756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.659723 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" event={"ID":"1c4f99bf-f802-4ae9-b736-f19e0f12af98","Type":"ContainerStarted","Data":"c7eebc005ec2b278cbab5191195bc732ee04d01f084e6ee4c0d6c03c200ac6ff"} Oct 31 00:44:01.661455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.661434 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" event={"ID":"114ccd9b-8cfc-463c-8a72-5e5b2ea61105","Type":"ContainerStarted","Data":"ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a"} Oct 31 00:44:01.661569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.661464 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" event={"ID":"114ccd9b-8cfc-463c-8a72-5e5b2ea61105","Type":"ContainerStarted","Data":"283a9f30d41af7a98916ee54c0d13d5693b37acc2cd66a8d75ab011dab059d35"} Oct 31 00:44:01.661626 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.661566 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:01.682554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:01.682458 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" podStartSLOduration=1.682442848 podStartE2EDuration="1.682442848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:01.681348654 +0000 UTC m=+2725.742123350" watchObservedRunningTime="2026-10-31 00:44:01.682442848 +0000 UTC m=+2725.743217544" Oct 31 00:44:02.668394 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:02.668356 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" event={"ID":"1c4f99bf-f802-4ae9-b736-f19e0f12af98","Type":"ContainerStarted","Data":"98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d"} Oct 31 00:44:02.668769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:02.668407 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" event={"ID":"1c4f99bf-f802-4ae9-b736-f19e0f12af98","Type":"ContainerStarted","Data":"44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb"} Oct 31 00:44:02.695504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:02.695454 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" podStartSLOduration=2.695442641 podStartE2EDuration="2.695442641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:02.693225555 +0000 UTC m=+2726.754000249" watchObservedRunningTime="2026-10-31 00:44:02.695442641 +0000 UTC m=+2726.756217360" Oct 31 00:44:03.673179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:03.673145 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:03.673641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:03.673191 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:03.679355 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:03.679332 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:03.682100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:03.682080 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:44:10.949185 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:10.949143 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:12.659433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.659394 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.666985 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.666960 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.760019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.759995 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.760161 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.760114 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.763937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.763919 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") DevicePath \"csi-d4654be4ad7375db49ffa2409572339f1bb3740d7436b5c76259d4d03f9052a3\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.855078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.855049 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8fd5a5ac985486c6416e9f57ebb5ef27608808414a3869903174362f39d872c7/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.876865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.876834 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:12.918612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:12.918560 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:13.085816 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:44:13.085765 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:44:13.198695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:13.198672 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:44:13.201121 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:44:13.201096 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6262f9c4_3857_4e4d_b765_82e6618a25c4.slice/crio-7e9008f6943f118d5a927f79d46ea6530cb40ac66272459ffb5c04aa2072d766 WatchSource:0}: Error finding container 7e9008f6943f118d5a927f79d46ea6530cb40ac66272459ffb5c04aa2072d766: Status 404 returned error can't find the container with id 7e9008f6943f118d5a927f79d46ea6530cb40ac66272459ffb5c04aa2072d766 Oct 31 00:44:13.717945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:13.717907 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"6262f9c4-3857-4e4d-b765-82e6618a25c4","Type":"ContainerStarted","Data":"04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56"} Oct 31 00:44:13.718337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:13.717954 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"6262f9c4-3857-4e4d-b765-82e6618a25c4","Type":"ContainerStarted","Data":"7e9008f6943f118d5a927f79d46ea6530cb40ac66272459ffb5c04aa2072d766"} Oct 31 00:44:13.718337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:13.717978 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:13.741343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:13.741295 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podStartSLOduration=13.741284096 podStartE2EDuration="13.741284096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:13.739496547 +0000 UTC m=+2737.800271243" watchObservedRunningTime="2026-10-31 00:44:13.741284096 +0000 UTC m=+2737.802058791" Oct 31 00:44:20.949592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:20.949550 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:22.674506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:22.674480 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:44:30.949027 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:30.948976 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:34.728353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:34.728317 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:44:40.948575 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:40.948533 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:43.085841 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:44:43.085804 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:44:50.949122 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:44:50.949085 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:00.948490 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:00.948447 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:02.521455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.521423 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:45:02.522042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.521627 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podUID="6262f9c4-3857-4e4d-b765-82e6618a25c4" containerName="loki-index-gateway" containerID="cri-o://04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56" gracePeriod=30 Oct 31 00:45:02.601520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.601475 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth"] Oct 31 00:45:02.601720 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.601698 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" podUID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" containerName="loki-querier" containerID="cri-o://ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a" gracePeriod=30 Oct 31 00:45:02.612070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.612039 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp"] Oct 31 00:45:02.612269 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.612229 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="gateway" containerID="cri-o://44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb" gracePeriod=30 Oct 31 00:45:02.612361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.612230 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="opa" containerID="cri-o://98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d" gracePeriod=30 Oct 31 00:45:02.673002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.672963 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" podUID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" containerName="loki-querier" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:02.859020 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.858995 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:45:02.946104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.946068 5073 generic.go:356] "Generic (PLEG): container finished" podID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerID="98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d" exitCode=0 Oct 31 00:45:02.946291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.946137 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" event={"ID":"1c4f99bf-f802-4ae9-b736-f19e0f12af98","Type":"ContainerDied","Data":"98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d"} Oct 31 00:45:02.947363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.947341 5073 generic.go:356] "Generic (PLEG): container finished" podID="6262f9c4-3857-4e4d-b765-82e6618a25c4" containerID="04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56" exitCode=0 Oct 31 00:45:02.947484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.947390 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:45:02.947484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.947430 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"6262f9c4-3857-4e4d-b765-82e6618a25c4","Type":"ContainerDied","Data":"04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56"} Oct 31 00:45:02.947484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.947475 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"6262f9c4-3857-4e4d-b765-82e6618a25c4","Type":"ContainerDied","Data":"7e9008f6943f118d5a927f79d46ea6530cb40ac66272459ffb5c04aa2072d766"} Oct 31 00:45:02.947650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.947498 5073 scope.go:122] "RemoveContainer" containerID="04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56" Oct 31 00:45:02.956560 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.956545 5073 scope.go:122] "RemoveContainer" containerID="04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56" Oct 31 00:45:02.956840 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:02.956821 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56\": container with ID starting with 04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56 not found: ID does not exist" containerID="04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56" Oct 31 00:45:02.956895 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.956849 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56"} err="failed to get container status \"04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56\": rpc error: code = NotFound desc = could not find container \"04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56\": container with ID starting with 04032034f0be526b6885d6514be2ed4fd5c82489d7fe0dec2fcd16a20a84ea56 not found: ID does not exist" Oct 31 00:45:02.963072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963053 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963133 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963091 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6\" (UniqueName: \"kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963171 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963134 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963246 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963234 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963295 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963325 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963398 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle\") pod \"6262f9c4-3857-4e4d-b765-82e6618a25c4\" (UID: \"6262f9c4-3857-4e4d-b765-82e6618a25c4\") " Oct 31 00:45:02.963486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963420 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:02.963674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963638 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:02.963980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.963914 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:02.965870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.965740 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:02.965870 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.965849 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:02.966105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.966081 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:02.966105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.966085 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "kube-api-access-ftsj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:02.966537 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:02.966517 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42" pod "6262f9c4-3857-4e4d-b765-82e6618a25c4" (UID: "6262f9c4-3857-4e4d-b765-82e6618a25c4"). InnerVolumeSpecName "pvc-d40c9d92-b1bc-4286-b370-a2b061c99081". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:45:03.064462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.064429 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.064462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.064461 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsj6\" (UniqueName: \"kubernetes.io/projected/6262f9c4-3857-4e4d-b765-82e6618a25c4-kube-api-access-ftsj6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.064706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.064476 5073 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-test-loki-lifecycle-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.064706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.064512 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:45:03.064706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.064527 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.064706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.064542 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/6262f9c4-3857-4e4d-b765-82e6618a25c4-data-science-usage-logs-lokistack-index-gateway-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.074032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.074010 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-d40c9d92-b1bc-4286-b370-a2b061c99081" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:45:03.156770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.156744 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:45:03.165400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.165373 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-d4654be4ad7375db49ffa2409572339f1bb3740d7436b5c76259d4d03f9052a3\"" Oct 31 00:45:03.265904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.265872 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc\") pod \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " Oct 31 00:45:03.266058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.265916 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config\") pod \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " Oct 31 00:45:03.266058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.265971 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle\") pod \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " Oct 31 00:45:03.266152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266121 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw\" (UniqueName: \"kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw\") pod \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " Oct 31 00:45:03.266206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266164 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http\") pod \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " Oct 31 00:45:03.266206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266186 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret\") pod \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\" (UID: \"114ccd9b-8cfc-463c-8a72-5e5b2ea61105\") " Oct 31 00:45:03.266401 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266371 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config" pod "114ccd9b-8cfc-463c-8a72-5e5b2ea61105" (UID: "114ccd9b-8cfc-463c-8a72-5e5b2ea61105"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.266401 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266383 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle" pod "114ccd9b-8cfc-463c-8a72-5e5b2ea61105" (UID: "114ccd9b-8cfc-463c-8a72-5e5b2ea61105"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.266564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266539 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.266564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.266555 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.269120 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.269093 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc" pod "114ccd9b-8cfc-463c-8a72-5e5b2ea61105" (UID: "114ccd9b-8cfc-463c-8a72-5e5b2ea61105"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.269120 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.269104 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw" pod "114ccd9b-8cfc-463c-8a72-5e5b2ea61105" (UID: "114ccd9b-8cfc-463c-8a72-5e5b2ea61105"). InnerVolumeSpecName "kube-api-access-4jsqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.269249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.269147 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret" pod "114ccd9b-8cfc-463c-8a72-5e5b2ea61105" (UID: "114ccd9b-8cfc-463c-8a72-5e5b2ea61105"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.269430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.269401 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http" pod "114ccd9b-8cfc-463c-8a72-5e5b2ea61105" (UID: "114ccd9b-8cfc-463c-8a72-5e5b2ea61105"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.270485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.270465 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:45:03.274034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.274016 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:45:03.367343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.367309 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jsqw\" (UniqueName: \"kubernetes.io/projected/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-kube-api-access-4jsqw\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.367343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.367338 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.367343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.367349 5073 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-test-loki-lifecycle-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.367581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.367360 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/114ccd9b-8cfc-463c-8a72-5e5b2ea61105-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.673845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.673808 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="gateway" probeResult="failure" output="Get \"https://10.132.0.193:8081/ready\": dial tcp 10.132.0.193:8081: connect: connection refused" Oct 31 00:45:03.674245 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.673808 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="opa" probeResult="failure" output="Get \"https://10.132.0.193:8083/ready\": dial tcp 10.132.0.193:8083: connect: connection refused" Oct 31 00:45:03.869272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.869236 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:45:03.954424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.954391 5073 generic.go:356] "Generic (PLEG): container finished" podID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" containerID="ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a" exitCode=0 Oct 31 00:45:03.954592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.954442 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" Oct 31 00:45:03.954592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.954475 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" event={"ID":"114ccd9b-8cfc-463c-8a72-5e5b2ea61105","Type":"ContainerDied","Data":"ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a"} Oct 31 00:45:03.954592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.954515 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth" event={"ID":"114ccd9b-8cfc-463c-8a72-5e5b2ea61105","Type":"ContainerDied","Data":"283a9f30d41af7a98916ee54c0d13d5693b37acc2cd66a8d75ab011dab059d35"} Oct 31 00:45:03.954592 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.954534 5073 scope.go:122] "RemoveContainer" containerID="ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a" Oct 31 00:45:03.956692 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.956667 5073 generic.go:356] "Generic (PLEG): container finished" podID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerID="44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb" exitCode=0 Oct 31 00:45:03.956845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.956698 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" event={"ID":"1c4f99bf-f802-4ae9-b736-f19e0f12af98","Type":"ContainerDied","Data":"44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb"} Oct 31 00:45:03.956845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.956708 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" Oct 31 00:45:03.956845 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.956725 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp" event={"ID":"1c4f99bf-f802-4ae9-b736-f19e0f12af98","Type":"ContainerDied","Data":"c7eebc005ec2b278cbab5191195bc732ee04d01f084e6ee4c0d6c03c200ac6ff"} Oct 31 00:45:03.964008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.963990 5073 scope.go:122] "RemoveContainer" containerID="ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a" Oct 31 00:45:03.964272 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:03.964254 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a\": container with ID starting with ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a not found: ID does not exist" containerID="ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a" Oct 31 00:45:03.964326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.964280 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a"} err="failed to get container status \"ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a\": rpc error: code = NotFound desc = could not find container \"ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a\": container with ID starting with ab532ac1423feacbb427fa0dc3f0d7f3005fe020bc86def4c7597f21fcdba32a not found: ID does not exist" Oct 31 00:45:03.964326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.964296 5073 scope.go:122] "RemoveContainer" containerID="98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d" Oct 31 00:45:03.971977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.971958 5073 scope.go:122] "RemoveContainer" containerID="44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb" Oct 31 00:45:03.972299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972282 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972315 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972381 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972349 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972399 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972459 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972581 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972525 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972581 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.972619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.972608 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle\") pod \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\" (UID: \"1c4f99bf-f802-4ae9-b736-f19e0f12af98\") " Oct 31 00:45:03.973555 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.973376 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.976272 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.976246 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "kube-api-access-ltszv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.976362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.976307 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.978325 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.978301 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth"] Oct 31 00:45:03.978797 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.978749 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.978907 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.978822 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.979380 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.979344 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.981010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.980990 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.981740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.981721 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants" pod "1c4f99bf-f802-4ae9-b736-f19e0f12af98" (UID: "1c4f99bf-f802-4ae9-b736-f19e0f12af98"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.982165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.982144 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-nhrth"] Oct 31 00:45:03.991896 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.991876 5073 scope.go:122] "RemoveContainer" containerID="98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d" Oct 31 00:45:03.992219 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:03.992201 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d\": container with ID starting with 98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d not found: ID does not exist" containerID="98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d" Oct 31 00:45:03.992274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.992228 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d"} err="failed to get container status \"98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d\": rpc error: code = NotFound desc = could not find container \"98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d\": container with ID starting with 98d140d34be7f831dd80af256b653f913112d1bc4c3b9e18576791818144632d not found: ID does not exist" Oct 31 00:45:03.992274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.992245 5073 scope.go:122] "RemoveContainer" containerID="44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb" Oct 31 00:45:03.992523 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:03.992506 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb\": container with ID starting with 44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb not found: ID does not exist" containerID="44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb" Oct 31 00:45:03.992579 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:03.992527 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb"} err="failed to get container status \"44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb\": rpc error: code = NotFound desc = could not find container \"44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb\": container with ID starting with 44a99092e77dfcb64e5f083c6d279c288de09e2460e437d982d9ee12cfc06ecb not found: ID does not exist" Oct 31 00:45:04.058283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.058239 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" path="/var/lib/kubelet/pods/114ccd9b-8cfc-463c-8a72-5e5b2ea61105/volumes" Oct 31 00:45:04.058669 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.058656 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6262f9c4-3857-4e4d-b765-82e6618a25c4" path="/var/lib/kubelet/pods/6262f9c4-3857-4e4d-b765-82e6618a25c4/volumes" Oct 31 00:45:04.073833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073807 5073 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-rbac\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.073833 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073834 5073 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tenants\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.074002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073847 5073 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-tls-secret\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.074002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073861 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.074002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073875 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltszv\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-kube-api-access-ltszv\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.074002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073889 5073 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/1c4f99bf-f802-4ae9-b736-f19e0f12af98-lokistack-gateway\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.074002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073905 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.074002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.073920 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/1c4f99bf-f802-4ae9-b736-f19e0f12af98-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.274048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.274015 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp"] Oct 31 00:45:04.278733 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.278707 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-rschp"] Oct 31 00:45:04.775979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.775944 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:45:04.776440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:04.776135 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/seaweedfs" podUID="aa849695-ec53-4f11-b357-74e71e5afc91" containerName="seaweedfs" containerID="cri-o://a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7" gracePeriod=30 Oct 31 00:45:06.059738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:06.059703 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" path="/var/lib/kubelet/pods/1c4f99bf-f802-4ae9-b736-f19e0f12af98/volumes" Oct 31 00:45:10.949326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:10.949282 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:13.085203 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:13.085167 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:45:20.949162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:20.949122 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:30.037136 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.037080 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:45:30.072661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.072631 5073 generic.go:356] "Generic (PLEG): container finished" podID="aa849695-ec53-4f11-b357-74e71e5afc91" containerID="a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7" exitCode=0 Oct 31 00:45:30.072828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.072673 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"aa849695-ec53-4f11-b357-74e71e5afc91","Type":"ContainerDied","Data":"a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7"} Oct 31 00:45:30.072828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.072713 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"aa849695-ec53-4f11-b357-74e71e5afc91","Type":"ContainerDied","Data":"485726ff70ba33189e7614253c83c2023849da73ad9a3cbe9bdd70b5d2454ecd"} Oct 31 00:45:30.072828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.072736 5073 scope.go:122] "RemoveContainer" containerID="a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7" Oct 31 00:45:30.072828 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.072741 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:45:30.073691 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.073666 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data\" (UniqueName: \"kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data\") pod \"aa849695-ec53-4f11-b357-74e71e5afc91\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " Oct 31 00:45:30.073942 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.073925 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf\" (UniqueName: \"kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf\") pod \"aa849695-ec53-4f11-b357-74e71e5afc91\" (UID: \"aa849695-ec53-4f11-b357-74e71e5afc91\") " Oct 31 00:45:30.074976 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.074951 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data" pod "aa849695-ec53-4f11-b357-74e71e5afc91" (UID: "aa849695-ec53-4f11-b357-74e71e5afc91"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:30.076273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.076249 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf" pod "aa849695-ec53-4f11-b357-74e71e5afc91" (UID: "aa849695-ec53-4f11-b357-74e71e5afc91"). InnerVolumeSpecName "kube-api-access-jswwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:30.090053 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.090036 5073 scope.go:122] "RemoveContainer" containerID="a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7" Oct 31 00:45:30.090334 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:30.090318 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7\": container with ID starting with a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7 not found: ID does not exist" containerID="a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7" Oct 31 00:45:30.090384 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.090343 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7"} err="failed to get container status \"a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7\": rpc error: code = NotFound desc = could not find container \"a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7\": container with ID starting with a0375d2b7eccc9905faf8170a06d19c3b1663363c5b645fec1816339b408c9d7 not found: ID does not exist" Oct 31 00:45:30.175610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.175568 5073 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa849695-ec53-4f11-b357-74e71e5afc91-data\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:30.175610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.175608 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswwf\" (UniqueName: \"kubernetes.io/projected/aa849695-ec53-4f11-b357-74e71e5afc91-kube-api-access-jswwf\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:30.394236 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.394208 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:45:30.398060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.398038 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:45:30.948991 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:30.948956 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:32.059087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:32.059054 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa849695-ec53-4f11-b357-74e71e5afc91" path="/var/lib/kubelet/pods/aa849695-ec53-4f11-b357-74e71e5afc91/volumes" Oct 31 00:45:37.626299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626200 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887"] Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626552 5073 cpu_manager.go:414] "containerMap: removing container" podUID="aa849695-ec53-4f11-b357-74e71e5afc91" containerName="seaweedfs" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626569 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa849695-ec53-4f11-b357-74e71e5afc91" containerName="seaweedfs" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626584 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6262f9c4-3857-4e4d-b765-82e6618a25c4" containerName="loki-index-gateway" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626593 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6262f9c4-3857-4e4d-b765-82e6618a25c4" containerName="loki-index-gateway" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626618 5073 cpu_manager.go:414] "containerMap: removing container" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="opa" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626625 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="opa" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626649 5073 cpu_manager.go:414] "containerMap: removing container" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="gateway" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626657 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="gateway" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626683 5073 cpu_manager.go:414] "containerMap: removing container" podUID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" containerName="loki-querier" Oct 31 00:45:37.626701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626691 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" containerName="loki-querier" Oct 31 00:45:37.627067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626805 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="gateway" Oct 31 00:45:37.627067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626822 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="114ccd9b-8cfc-463c-8a72-5e5b2ea61105" containerName="loki-querier" Oct 31 00:45:37.627067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626832 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa849695-ec53-4f11-b357-74e71e5afc91" containerName="seaweedfs" Oct 31 00:45:37.627067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626840 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6262f9c4-3857-4e4d-b765-82e6618a25c4" containerName="loki-index-gateway" Oct 31 00:45:37.627067 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.626853 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4f99bf-f802-4ae9-b736-f19e0f12af98" containerName="opa" Oct 31 00:45:37.631289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.631271 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:37.637028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.636990 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:45:37.638822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.638265 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887"] Oct 31 00:45:37.732435 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.732409 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7b4q\" (UniqueName: \"kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887\" (UID: \"a4e6e99b-c919-4a9f-bc31-115046d65f2f\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:37.833511 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.833485 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7b4q\" (UniqueName: \"kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887\" (UID: \"a4e6e99b-c919-4a9f-bc31-115046d65f2f\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:37.846441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.846409 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7b4q\" (UniqueName: \"kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887\" (UID: \"a4e6e99b-c919-4a9f-bc31-115046d65f2f\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:37.945195 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:37.945124 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:38.008385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.008345 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb"] Oct 31 00:45:38.014958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.014933 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.026488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.025210 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb"] Oct 31 00:45:38.034229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.033572 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:45:38.034229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.033693 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:45:38.034229 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.033819 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-vc59r\"" Oct 31 00:45:38.034894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.034865 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95jz\" (UniqueName: \"kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.035002 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.034933 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.035069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.035045 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.135932 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.135900 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95jz\" (UniqueName: \"kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.136101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.135945 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.136101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.136014 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.136767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.136690 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.138550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.138528 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.147940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.147915 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95jz\" (UniqueName: \"kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz\") pod \"data-science-collector-targetallocator-d66d74696-lthtb\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.233625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.233603 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887"] Oct 31 00:45:38.236602 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:45:38.236578 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e6e99b_c919_4a9f_bc31_115046d65f2f.slice/crio-2b2d9ece396ff801d14172b0911215818ec7e7baad0b73e74cee10e91684b1ac WatchSource:0}: Error finding container 2b2d9ece396ff801d14172b0911215818ec7e7baad0b73e74cee10e91684b1ac: Status 404 returned error can't find the container with id 2b2d9ece396ff801d14172b0911215818ec7e7baad0b73e74cee10e91684b1ac Oct 31 00:45:38.341690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.341654 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:38.619142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.619069 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb"] Oct 31 00:45:38.622847 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:45:38.622815 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf625ee5d_9203_4a69_a48b_8057a4c59fd5.slice/crio-f714070e59b9fe5885439241414e3d4fe37300e9aece61a63c671fa7035e330c WatchSource:0}: Error finding container f714070e59b9fe5885439241414e3d4fe37300e9aece61a63c671fa7035e330c: Status 404 returned error can't find the container with id f714070e59b9fe5885439241414e3d4fe37300e9aece61a63c671fa7035e330c Oct 31 00:45:38.624872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.624838 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:45:38.624967 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:38.624896 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:45:39.115044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.115008 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" event={"ID":"a4e6e99b-c919-4a9f-bc31-115046d65f2f","Type":"ContainerStarted","Data":"e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9"} Oct 31 00:45:39.115044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.115051 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" event={"ID":"a4e6e99b-c919-4a9f-bc31-115046d65f2f","Type":"ContainerStarted","Data":"2b2d9ece396ff801d14172b0911215818ec7e7baad0b73e74cee10e91684b1ac"} Oct 31 00:45:39.116539 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.116510 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" event={"ID":"f625ee5d-9203-4a69-a48b-8057a4c59fd5","Type":"ContainerStarted","Data":"ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5"} Oct 31 00:45:39.116539 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.116547 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" event={"ID":"f625ee5d-9203-4a69-a48b-8057a4c59fd5","Type":"ContainerStarted","Data":"f714070e59b9fe5885439241414e3d4fe37300e9aece61a63c671fa7035e330c"} Oct 31 00:45:39.116730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.116638 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:39.118252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.118227 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:45:39.133194 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.133157 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" podStartSLOduration=2.13314612 podStartE2EDuration="2.13314612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:39.132062838 +0000 UTC m=+2823.192837534" watchObservedRunningTime="2026-10-31 00:45:39.13314612 +0000 UTC m=+2823.193920814" Oct 31 00:45:39.152636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:39.152589 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" podStartSLOduration=2.152562449 podStartE2EDuration="2.152562449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:39.152186077 +0000 UTC m=+2823.212960773" watchObservedRunningTime="2026-10-31 00:45:39.152562449 +0000 UTC m=+2823.213337144" Oct 31 00:45:40.949461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:40.949426 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:43.084628 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:43.084593 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:45:43.444878 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.444799 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl"] Oct 31 00:45:43.448415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.448391 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:45:43.457215 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.456992 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl"] Oct 31 00:45:43.480156 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.480129 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgx44\" (UniqueName: \"kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl\" (UID: \"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:45:43.581268 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.581240 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgx44\" (UniqueName: \"kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl\" (UID: \"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:45:43.593450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.593426 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgx44\" (UniqueName: \"kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl\" (UID: \"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:45:43.761019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:43.760999 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:45:44.036966 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.036897 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl"] Oct 31 00:45:44.040256 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:45:44.040227 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98c62573_128b_46b3_b6e8_7b9b1f3b5bdb.slice/crio-6fbb0d988ae1052482ef90d338e1eb20073c65f377cba78f92bf41ce4611c131 WatchSource:0}: Error finding container 6fbb0d988ae1052482ef90d338e1eb20073c65f377cba78f92bf41ce4611c131: Status 404 returned error can't find the container with id 6fbb0d988ae1052482ef90d338e1eb20073c65f377cba78f92bf41ce4611c131 Oct 31 00:45:44.139430 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.139398 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" event={"ID":"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb","Type":"ContainerStarted","Data":"f2b9e4fbcafff6beec9b5eb87a3f0c8d9c2cc21e57b22b5ccd96997c7450b0f2"} Oct 31 00:45:44.139779 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.139437 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" event={"ID":"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb","Type":"ContainerStarted","Data":"6fbb0d988ae1052482ef90d338e1eb20073c65f377cba78f92bf41ce4611c131"} Oct 31 00:45:44.156322 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.156276 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" podStartSLOduration=1.156263262 podStartE2EDuration="1.156263262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:44.15535571 +0000 UTC m=+2828.216130408" watchObservedRunningTime="2026-10-31 00:45:44.156263262 +0000 UTC m=+2828.217037956" Oct 31 00:45:44.184336 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.184312 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887"] Oct 31 00:45:44.184476 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.184458 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" podUID="a4e6e99b-c919-4a9f-bc31-115046d65f2f" containerName="thanos-querier" containerID="cri-o://e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9" gracePeriod=30 Oct 31 00:45:44.272012 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.271979 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:45:44.275757 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.275735 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.285585 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.285565 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:45:44.285684 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.285670 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:45:44.285746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.285735 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:45:44.298338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.298166 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:45:44.372008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.371976 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:45:44.372150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.372015 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:45:44.372150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.372015 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:45:44.372150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.372042 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:45:44.372150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.372015 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:45:44.372150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.372081 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:45:44.372150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.372100 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b6hd7\"" Oct 31 00:45:44.388292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388270 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388302 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388390 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388434 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388426 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388494 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388539 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388572 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388591 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388629 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388709 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388681 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr2d\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388743 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388709 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.388821 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.388808 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.420707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.420688 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:44.490013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.489988 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q\" (UniqueName: \"kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q\") pod \"a4e6e99b-c919-4a9f-bc31-115046d65f2f\" (UID: \"a4e6e99b-c919-4a9f-bc31-115046d65f2f\") " Oct 31 00:45:44.490128 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490108 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490133 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490176 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490154 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490289 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490342 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490374 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490408 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490459 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490452 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490481 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490515 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490580 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr2d\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.490715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.490607 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.491566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.491539 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.491656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.491620 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.491892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.491861 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.493100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.493071 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q" pod "a4e6e99b-c919-4a9f-bc31-115046d65f2f" (UID: "a4e6e99b-c919-4a9f-bc31-115046d65f2f"). InnerVolumeSpecName "kube-api-access-c7b4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:44.493714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.493688 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.493835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.493729 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.494190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.494168 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.494315 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.494290 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.494528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.494502 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.494566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.494521 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.494917 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.494900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.495288 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.495271 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.503763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.503741 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr2d\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:44.591113 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:44.591062 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7b4q\" (UniqueName: \"kubernetes.io/projected/a4e6e99b-c919-4a9f-bc31-115046d65f2f-kube-api-access-c7b4q\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:45.144861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.144829 5073 generic.go:356] "Generic (PLEG): container finished" podID="a4e6e99b-c919-4a9f-bc31-115046d65f2f" containerID="e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9" exitCode=0 Oct 31 00:45:45.145308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.144869 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" Oct 31 00:45:45.145308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.144912 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" event={"ID":"a4e6e99b-c919-4a9f-bc31-115046d65f2f","Type":"ContainerDied","Data":"e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9"} Oct 31 00:45:45.145308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.144940 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887" event={"ID":"a4e6e99b-c919-4a9f-bc31-115046d65f2f","Type":"ContainerDied","Data":"2b2d9ece396ff801d14172b0911215818ec7e7baad0b73e74cee10e91684b1ac"} Oct 31 00:45:45.145308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.144955 5073 scope.go:122] "RemoveContainer" containerID="e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9" Oct 31 00:45:45.154274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.154248 5073 scope.go:122] "RemoveContainer" containerID="e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9" Oct 31 00:45:45.154530 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:45:45.154512 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9\": container with ID starting with e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9 not found: ID does not exist" containerID="e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9" Oct 31 00:45:45.154577 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.154538 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9"} err="failed to get container status \"e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9\": rpc error: code = NotFound desc = could not find container \"e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9\": container with ID starting with e07dcb6d60071962c5409018de719fdcfdbbd21d3db4712deda9d1e48c4fdea9 not found: ID does not exist" Oct 31 00:45:45.165668 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.165647 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887"] Oct 31 00:45:45.169699 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:45.169675 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-c7887"] Oct 31 00:45:46.058841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:46.058809 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e6e99b-c919-4a9f-bc31-115046d65f2f" path="/var/lib/kubelet/pods/a4e6e99b-c919-4a9f-bc31-115046d65f2f/volumes" Oct 31 00:45:50.949892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:50.949854 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:52.556746 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.556710 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.563378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.563351 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.657167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.657140 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.657298 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.657278 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.660703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.660686 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.753127 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.753104 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.768529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.768505 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:52.996466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:52.996440 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:45:53.287009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:53.286983 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:45:53.288306 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:45:53.288279 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c7dec4_84e2_4e7d_ad60_9bca21493b43.slice/crio-3325c11e1ddca6128f2120088111e4599021e5538879df755779a298c4b9e1f8 WatchSource:0}: Error finding container 3325c11e1ddca6128f2120088111e4599021e5538879df755779a298c4b9e1f8: Status 404 returned error can't find the container with id 3325c11e1ddca6128f2120088111e4599021e5538879df755779a298c4b9e1f8 Oct 31 00:45:54.188903 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:54.188873 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerStarted","Data":"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a"} Oct 31 00:45:54.189289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:54.188930 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerStarted","Data":"3325c11e1ddca6128f2120088111e4599021e5538879df755779a298c4b9e1f8"} Oct 31 00:45:58.206968 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:58.206937 5073 generic.go:356] "Generic (PLEG): container finished" podID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerID="ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a" exitCode=0 Oct 31 00:45:58.207396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:58.206984 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerDied","Data":"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a"} Oct 31 00:45:59.214345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:59.214309 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerStarted","Data":"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e"} Oct 31 00:45:59.214715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:59.214380 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerStarted","Data":"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b"} Oct 31 00:45:59.214715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:59.214409 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerStarted","Data":"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61"} Oct 31 00:45:59.246584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:45:59.246536 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=11.462762138 podStartE2EDuration="15.246525601s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783763463s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:59.245422605 +0000 UTC m=+2843.306197300" watchObservedRunningTime="2026-10-31 00:45:59.246525601 +0000 UTC m=+2843.307300295" Oct 31 00:46:00.949279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:00.949242 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:02.997284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:02.997252 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:46:07.997209 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:07.997176 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:46:08.002854 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:08.002826 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:46:08.257858 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:08.257773 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:46:10.948922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:10.948887 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:13.085378 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:46:13.085344 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:46:17.994078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:17.994043 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg"] Oct 31 00:46:17.994494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:17.994484 5073 cpu_manager.go:414] "containerMap: removing container" podUID="a4e6e99b-c919-4a9f-bc31-115046d65f2f" containerName="thanos-querier" Oct 31 00:46:17.994543 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:17.994498 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e6e99b-c919-4a9f-bc31-115046d65f2f" containerName="thanos-querier" Oct 31 00:46:17.994587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:17.994570 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e6e99b-c919-4a9f-bc31-115046d65f2f" containerName="thanos-querier" Oct 31 00:46:17.997965 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:17.997939 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.011894 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.011868 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg"] Oct 31 00:46:18.155748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.155719 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.155905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.155750 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndlg\" (UniqueName: \"kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.155905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.155877 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.256680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.256620 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.256680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.256672 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.256858 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.256800 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndlg\" (UniqueName: \"kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.257259 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.257238 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.259260 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.259230 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.269276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.269253 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndlg\" (UniqueName: \"kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg\") pod \"data-science-collector-targetallocator-6c57b97bc4-lqffg\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.309448 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.309425 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:18.587137 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.587042 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg"] Oct 31 00:46:18.589940 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:46:18.589906 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55f836bc_07c6_4b21_b1bf_c1d619e8c586.slice/crio-2ba75d89b98cb671c1304d811cd7eb16f7a5a838d6a95d3449d37fd5969c4bf1 WatchSource:0}: Error finding container 2ba75d89b98cb671c1304d811cd7eb16f7a5a838d6a95d3449d37fd5969c4bf1: Status 404 returned error can't find the container with id 2ba75d89b98cb671c1304d811cd7eb16f7a5a838d6a95d3449d37fd5969c4bf1 Oct 31 00:46:18.592228 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.592196 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:46:18.592296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:18.592269 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:46:19.307036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.306994 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" event={"ID":"55f836bc-07c6-4b21-b1bf-c1d619e8c586","Type":"ContainerStarted","Data":"60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10"} Oct 31 00:46:19.307036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.307041 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" event={"ID":"55f836bc-07c6-4b21-b1bf-c1d619e8c586","Type":"ContainerStarted","Data":"2ba75d89b98cb671c1304d811cd7eb16f7a5a838d6a95d3449d37fd5969c4bf1"} Oct 31 00:46:19.307465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.307058 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:19.308903 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.308878 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:19.327296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.327239 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" podStartSLOduration=2.327226989 podStartE2EDuration="2.327226989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:19.325944417 +0000 UTC m=+2863.386719113" watchObservedRunningTime="2026-10-31 00:46:19.327226989 +0000 UTC m=+2863.388001747" Oct 31 00:46:19.376106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.376079 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb"] Oct 31 00:46:19.376279 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.376255 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" podUID="f625ee5d-9203-4a69-a48b-8057a4c59fd5" containerName="ta-container" containerID="cri-o://ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5" gracePeriod=30 Oct 31 00:46:19.634275 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.634237 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:46:19.773318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.773282 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert\") pod \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " Oct 31 00:46:19.773501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.773374 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz\" (UniqueName: \"kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz\") pod \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " Oct 31 00:46:19.773501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.773431 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal\" (UniqueName: \"kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal\") pod \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\" (UID: \"f625ee5d-9203-4a69-a48b-8057a4c59fd5\") " Oct 31 00:46:19.773822 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.773779 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal" pod "f625ee5d-9203-4a69-a48b-8057a4c59fd5" (UID: "f625ee5d-9203-4a69-a48b-8057a4c59fd5"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:19.775471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.775450 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert" pod "f625ee5d-9203-4a69-a48b-8057a4c59fd5" (UID: "f625ee5d-9203-4a69-a48b-8057a4c59fd5"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.775611 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.775589 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz" pod "f625ee5d-9203-4a69-a48b-8057a4c59fd5" (UID: "f625ee5d-9203-4a69-a48b-8057a4c59fd5"). InnerVolumeSpecName "kube-api-access-f95jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:19.875099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.875036 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f625ee5d-9203-4a69-a48b-8057a4c59fd5-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.875099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.875063 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f95jz\" (UniqueName: \"kubernetes.io/projected/f625ee5d-9203-4a69-a48b-8057a4c59fd5-kube-api-access-f95jz\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.875099 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:19.875075 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f625ee5d-9203-4a69-a48b-8057a4c59fd5-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:20.312809 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.312750 5073 generic.go:356] "Generic (PLEG): container finished" podID="f625ee5d-9203-4a69-a48b-8057a4c59fd5" containerID="ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5" exitCode=0 Oct 31 00:46:20.313281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.312827 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" Oct 31 00:46:20.313281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.312836 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" event={"ID":"f625ee5d-9203-4a69-a48b-8057a4c59fd5","Type":"ContainerDied","Data":"ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5"} Oct 31 00:46:20.313281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.312880 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb" event={"ID":"f625ee5d-9203-4a69-a48b-8057a4c59fd5","Type":"ContainerDied","Data":"f714070e59b9fe5885439241414e3d4fe37300e9aece61a63c671fa7035e330c"} Oct 31 00:46:20.313281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.312899 5073 scope.go:122] "RemoveContainer" containerID="ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5" Oct 31 00:46:20.321662 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.321649 5073 scope.go:122] "RemoveContainer" containerID="ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5" Oct 31 00:46:20.321965 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:46:20.321942 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5\": container with ID starting with ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5 not found: ID does not exist" containerID="ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5" Oct 31 00:46:20.322073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.321973 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5"} err="failed to get container status \"ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5\": rpc error: code = NotFound desc = could not find container \"ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5\": container with ID starting with ce81df41fca207a1e9bd97915583d9dd6dbb3426f4db0eb6d7d1932cab3575b5 not found: ID does not exist" Oct 31 00:46:20.330377 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.330356 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb"] Oct 31 00:46:20.334842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.334821 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-d66d74696-lthtb"] Oct 31 00:46:20.949083 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:20.949046 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:22.062799 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:22.062738 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f625ee5d-9203-4a69-a48b-8057a4c59fd5" path="/var/lib/kubelet/pods/f625ee5d-9203-4a69-a48b-8057a4c59fd5/volumes" Oct 31 00:46:30.076625 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.076596 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d"] Oct 31 00:46:30.077147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.076971 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f625ee5d-9203-4a69-a48b-8057a4c59fd5" containerName="ta-container" Oct 31 00:46:30.077147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.076985 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f625ee5d-9203-4a69-a48b-8057a4c59fd5" containerName="ta-container" Oct 31 00:46:30.077147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.077076 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f625ee5d-9203-4a69-a48b-8057a4c59fd5" containerName="ta-container" Oct 31 00:46:30.081535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.081513 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.092025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.091998 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d"] Oct 31 00:46:30.157164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.157138 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhg6\" (UniqueName: \"kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.157324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.157191 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.157324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.157211 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.258155 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.258119 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.258308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.258166 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.258308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.258293 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhg6\" (UniqueName: \"kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.258750 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.258729 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.260830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.260809 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.271329 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.271300 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhg6\" (UniqueName: \"kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6\") pod \"data-science-collector-targetallocator-6b8d956fc5-h894d\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.395695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.395635 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:30.671112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.671088 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d"] Oct 31 00:46:30.673876 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:46:30.673832 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffaa97a2_1a3b_4900_9502_d997146cd809.slice/crio-8bec5d3b23f4871cc6dc06a465b18df153f3b23333121e514209e57f104c49fd WatchSource:0}: Error finding container 8bec5d3b23f4871cc6dc06a465b18df153f3b23333121e514209e57f104c49fd: Status 404 returned error can't find the container with id 8bec5d3b23f4871cc6dc06a465b18df153f3b23333121e514209e57f104c49fd Oct 31 00:46:30.675961 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.675927 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:46:30.676060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.675998 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:46:30.949218 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:30.949131 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:31.365089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.365053 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" event={"ID":"ffaa97a2-1a3b-4900-9502-d997146cd809","Type":"ContainerStarted","Data":"f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c"} Oct 31 00:46:31.365495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.365098 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" event={"ID":"ffaa97a2-1a3b-4900-9502-d997146cd809","Type":"ContainerStarted","Data":"8bec5d3b23f4871cc6dc06a465b18df153f3b23333121e514209e57f104c49fd"} Oct 31 00:46:31.365495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.365202 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:31.366980 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.366955 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:46:31.385528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.385486 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" podStartSLOduration=1.385473639 podStartE2EDuration="1.385473639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:31.384150431 +0000 UTC m=+2875.444925130" watchObservedRunningTime="2026-10-31 00:46:31.385473639 +0000 UTC m=+2875.446248333" Oct 31 00:46:31.434277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.434243 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg"] Oct 31 00:46:31.434457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.434432 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" podUID="55f836bc-07c6-4b21-b1bf-c1d619e8c586" containerName="ta-container" containerID="cri-o://60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10" gracePeriod=30 Oct 31 00:46:31.696562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.696539 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:31.773197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.773170 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg\" (UniqueName: \"kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg\") pod \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " Oct 31 00:46:31.773360 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.773212 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert\") pod \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " Oct 31 00:46:31.773433 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.773352 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal\" (UniqueName: \"kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal\") pod \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\" (UID: \"55f836bc-07c6-4b21-b1bf-c1d619e8c586\") " Oct 31 00:46:31.773680 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.773659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal" pod "55f836bc-07c6-4b21-b1bf-c1d619e8c586" (UID: "55f836bc-07c6-4b21-b1bf-c1d619e8c586"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:31.775437 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.775416 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert" pod "55f836bc-07c6-4b21-b1bf-c1d619e8c586" (UID: "55f836bc-07c6-4b21-b1bf-c1d619e8c586"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:31.775522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.775508 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg" pod "55f836bc-07c6-4b21-b1bf-c1d619e8c586" (UID: "55f836bc-07c6-4b21-b1bf-c1d619e8c586"). InnerVolumeSpecName "kube-api-access-nndlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:31.874028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.874000 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/55f836bc-07c6-4b21-b1bf-c1d619e8c586-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:31.874028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.874022 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndlg\" (UniqueName: \"kubernetes.io/projected/55f836bc-07c6-4b21-b1bf-c1d619e8c586-kube-api-access-nndlg\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:31.874028 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:31.874032 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/55f836bc-07c6-4b21-b1bf-c1d619e8c586-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.370653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.370569 5073 generic.go:356] "Generic (PLEG): container finished" podID="55f836bc-07c6-4b21-b1bf-c1d619e8c586" containerID="60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10" exitCode=0 Oct 31 00:46:32.370653 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.370643 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" Oct 31 00:46:32.371203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.370650 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" event={"ID":"55f836bc-07c6-4b21-b1bf-c1d619e8c586","Type":"ContainerDied","Data":"60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10"} Oct 31 00:46:32.371203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.370697 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg" event={"ID":"55f836bc-07c6-4b21-b1bf-c1d619e8c586","Type":"ContainerDied","Data":"2ba75d89b98cb671c1304d811cd7eb16f7a5a838d6a95d3449d37fd5969c4bf1"} Oct 31 00:46:32.371203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.370714 5073 scope.go:122] "RemoveContainer" containerID="60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10" Oct 31 00:46:32.379560 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.379535 5073 scope.go:122] "RemoveContainer" containerID="60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10" Oct 31 00:46:32.379862 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:46:32.379846 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10\": container with ID starting with 60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10 not found: ID does not exist" containerID="60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10" Oct 31 00:46:32.379905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.379870 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10"} err="failed to get container status \"60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10\": rpc error: code = NotFound desc = could not find container \"60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10\": container with ID starting with 60f8fa412f38b7c009e5eb1074b59ad7ba81093dc530ab3058c295935e167d10 not found: ID does not exist" Oct 31 00:46:32.391613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.391593 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg"] Oct 31 00:46:32.397131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:32.397111 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c57b97bc4-lqffg"] Oct 31 00:46:34.058372 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:34.058336 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f836bc-07c6-4b21-b1bf-c1d619e8c586" path="/var/lib/kubelet/pods/55f836bc-07c6-4b21-b1bf-c1d619e8c586/volumes" Oct 31 00:46:40.949607 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:40.949566 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:46:43.084886 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:46:43.084854 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:46:50.948599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:46:50.948559 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:00.949287 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:00.949250 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:03.788504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.788469 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl"] Oct 31 00:47:03.789144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.789076 5073 cpu_manager.go:414] "containerMap: removing container" podUID="55f836bc-07c6-4b21-b1bf-c1d619e8c586" containerName="ta-container" Oct 31 00:47:03.789144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.789098 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f836bc-07c6-4b21-b1bf-c1d619e8c586" containerName="ta-container" Oct 31 00:47:03.789306 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.789189 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f836bc-07c6-4b21-b1bf-c1d619e8c586" containerName="ta-container" Oct 31 00:47:03.792307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.792284 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:03.802112 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.802087 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl"] Oct 31 00:47:03.931056 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.931029 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:03.931199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.931069 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:03.931252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:03.931192 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w772d\" (UniqueName: \"kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.031852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.031824 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w772d\" (UniqueName: \"kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.031984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.031881 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.031984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.031921 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.032573 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.032541 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.034756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.034731 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.045383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.045328 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w772d\" (UniqueName: \"kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d\") pod \"data-science-collector-targetallocator-5d7496d9fb-l4qsl\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.104522 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.104494 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.379856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.379769 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl"] Oct 31 00:47:04.382035 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:47:04.382009 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7611cf43_05c1_4940_87a7_c66552ecdf99.slice/crio-3dbbc357394236e47a7a8b89c33405f439bb93ac1eb20fa59213584b478ee726 WatchSource:0}: Error finding container 3dbbc357394236e47a7a8b89c33405f439bb93ac1eb20fa59213584b478ee726: Status 404 returned error can't find the container with id 3dbbc357394236e47a7a8b89c33405f439bb93ac1eb20fa59213584b478ee726 Oct 31 00:47:04.384096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.384063 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:47:04.384173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.384122 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:47:04.520521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.520488 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" event={"ID":"7611cf43-05c1-4940-87a7-c66552ecdf99","Type":"ContainerStarted","Data":"3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6"} Oct 31 00:47:04.520695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.520528 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" event={"ID":"7611cf43-05c1-4940-87a7-c66552ecdf99","Type":"ContainerStarted","Data":"3dbbc357394236e47a7a8b89c33405f439bb93ac1eb20fa59213584b478ee726"} Oct 31 00:47:04.520695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.520612 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:04.521650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.521626 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" podUID="7611cf43-05c1-4940-87a7-c66552ecdf99" containerName="ta-container" probeResult="failure" output="Get \"http://10.132.0.200:8080/readyz\": dial tcp 10.132.0.200:8080: connect: connection refused" Oct 31 00:47:04.542971 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:04.542935 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" podStartSLOduration=1.542924598 podStartE2EDuration="1.542924598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:04.541531411 +0000 UTC m=+2908.602306106" watchObservedRunningTime="2026-10-31 00:47:04.542924598 +0000 UTC m=+2908.603699293" Oct 31 00:47:05.526846 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.526820 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:05.582962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.582931 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d"] Oct 31 00:47:05.583163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.583127 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" podUID="ffaa97a2-1a3b-4900-9502-d997146cd809" containerName="ta-container" containerID="cri-o://f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c" gracePeriod=30 Oct 31 00:47:05.850324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.850300 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:47:05.947599 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.947569 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal\" (UniqueName: \"kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal\") pod \"ffaa97a2-1a3b-4900-9502-d997146cd809\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " Oct 31 00:47:05.947766 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.947612 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert\") pod \"ffaa97a2-1a3b-4900-9502-d997146cd809\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " Oct 31 00:47:05.947766 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.947647 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6\" (UniqueName: \"kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6\") pod \"ffaa97a2-1a3b-4900-9502-d997146cd809\" (UID: \"ffaa97a2-1a3b-4900-9502-d997146cd809\") " Oct 31 00:47:05.948010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.947985 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal" pod "ffaa97a2-1a3b-4900-9502-d997146cd809" (UID: "ffaa97a2-1a3b-4900-9502-d997146cd809"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:05.949896 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.949868 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6" pod "ffaa97a2-1a3b-4900-9502-d997146cd809" (UID: "ffaa97a2-1a3b-4900-9502-d997146cd809"). InnerVolumeSpecName "kube-api-access-ldhg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:05.949989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:05.949898 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert" pod "ffaa97a2-1a3b-4900-9502-d997146cd809" (UID: "ffaa97a2-1a3b-4900-9502-d997146cd809"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:06.048605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.048581 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/ffaa97a2-1a3b-4900-9502-d997146cd809-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.048605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.048605 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/ffaa97a2-1a3b-4900-9502-d997146cd809-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.048740 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.048615 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhg6\" (UniqueName: \"kubernetes.io/projected/ffaa97a2-1a3b-4900-9502-d997146cd809-kube-api-access-ldhg6\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.531864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.531827 5073 generic.go:356] "Generic (PLEG): container finished" podID="ffaa97a2-1a3b-4900-9502-d997146cd809" containerID="f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c" exitCode=0 Oct 31 00:47:06.532367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.531873 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" Oct 31 00:47:06.532367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.531915 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" event={"ID":"ffaa97a2-1a3b-4900-9502-d997146cd809","Type":"ContainerDied","Data":"f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c"} Oct 31 00:47:06.532367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.531961 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d" event={"ID":"ffaa97a2-1a3b-4900-9502-d997146cd809","Type":"ContainerDied","Data":"8bec5d3b23f4871cc6dc06a465b18df153f3b23333121e514209e57f104c49fd"} Oct 31 00:47:06.532367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.531982 5073 scope.go:122] "RemoveContainer" containerID="f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c" Oct 31 00:47:06.540584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.540568 5073 scope.go:122] "RemoveContainer" containerID="f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c" Oct 31 00:47:06.540854 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:06.540839 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c\": container with ID starting with f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c not found: ID does not exist" containerID="f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c" Oct 31 00:47:06.540916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.540863 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c"} err="failed to get container status \"f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c\": rpc error: code = NotFound desc = could not find container \"f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c\": container with ID starting with f54eb998a787b68af4819c96b76b90430e79c06fb5b2053b1a604e0f851fe39c not found: ID does not exist" Oct 31 00:47:06.550214 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.550193 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d"] Oct 31 00:47:06.554338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:06.554318 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6b8d956fc5-h894d"] Oct 31 00:47:08.059411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:08.059382 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffaa97a2-1a3b-4900-9502-d997146cd809" path="/var/lib/kubelet/pods/ffaa97a2-1a3b-4900-9502-d997146cd809/volumes" Oct 31 00:47:09.428296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.428257 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl"] Oct 31 00:47:09.428767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.428473 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" podUID="98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" containerName="thanos-querier" containerID="cri-o://f2b9e4fbcafff6beec9b5eb87a3f0c8d9c2cc21e57b22b5ccd96997c7450b0f2" gracePeriod=30 Oct 31 00:47:09.517263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.517228 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:47:09.517554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.517518 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="prometheus" containerID="cri-o://0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" gracePeriod=600 Oct 31 00:47:09.517554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.517554 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="thanos-sidecar" containerID="cri-o://cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" gracePeriod=600 Oct 31 00:47:09.517875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.517555 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="config-reloader" containerID="cri-o://e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" gracePeriod=600 Oct 31 00:47:09.550024 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.549999 5073 generic.go:356] "Generic (PLEG): container finished" podID="98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" containerID="f2b9e4fbcafff6beec9b5eb87a3f0c8d9c2cc21e57b22b5ccd96997c7450b0f2" exitCode=0 Oct 31 00:47:09.550130 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.550073 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" event={"ID":"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb","Type":"ContainerDied","Data":"f2b9e4fbcafff6beec9b5eb87a3f0c8d9c2cc21e57b22b5ccd96997c7450b0f2"} Oct 31 00:47:09.714919 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.714886 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl"] Oct 31 00:47:09.715115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.715084 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" podUID="7611cf43-05c1-4940-87a7-c66552ecdf99" containerName="ta-container" containerID="cri-o://3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6" gracePeriod=30 Oct 31 00:47:09.880110 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.880076 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:47:09.981808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.981764 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44\" (UniqueName: \"kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44\") pod \"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb\" (UID: \"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb\") " Oct 31 00:47:09.984705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:09.984673 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44" pod "98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" (UID: "98c62573-128b-46b3-b6e8-7b9b1f3b5bdb"). InnerVolumeSpecName "kube-api-access-vgx44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.023244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.023214 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:10.083475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.083439 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgx44\" (UniqueName: \"kubernetes.io/projected/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb-kube-api-access-vgx44\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.183998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.183964 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal\" (UniqueName: \"kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal\") pod \"7611cf43-05c1-4940-87a7-c66552ecdf99\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " Oct 31 00:47:10.184239 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.184068 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert\") pod \"7611cf43-05c1-4940-87a7-c66552ecdf99\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " Oct 31 00:47:10.184239 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.184091 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d\" (UniqueName: \"kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d\") pod \"7611cf43-05c1-4940-87a7-c66552ecdf99\" (UID: \"7611cf43-05c1-4940-87a7-c66552ecdf99\") " Oct 31 00:47:10.184428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.184393 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal" pod "7611cf43-05c1-4940-87a7-c66552ecdf99" (UID: "7611cf43-05c1-4940-87a7-c66552ecdf99"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.186385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.186361 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert" pod "7611cf43-05c1-4940-87a7-c66552ecdf99" (UID: "7611cf43-05c1-4940-87a7-c66552ecdf99"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.186385 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.186374 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d" pod "7611cf43-05c1-4940-87a7-c66552ecdf99" (UID: "7611cf43-05c1-4940-87a7-c66552ecdf99"). InnerVolumeSpecName "kube-api-access-w772d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.290865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.286315 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/7611cf43-05c1-4940-87a7-c66552ecdf99-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.290865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.286356 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/7611cf43-05c1-4940-87a7-c66552ecdf99-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.290865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.286373 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w772d\" (UniqueName: \"kubernetes.io/projected/7611cf43-05c1-4940-87a7-c66552ecdf99-kube-api-access-w772d\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.320645 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.320620 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:10.387530 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387503 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387561 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387590 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387617 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387672 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387643 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387718 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387741 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387764 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out\" (UniqueName: \"kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387820 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387851 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387885 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387915 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.387992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.387969 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config\") pod \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\" (UID: \"09c7dec4-84e2-4e7d-ad60-9bca21493b43\") " Oct 31 00:47:10.388415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.388085 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.388415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.388108 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.388415 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.388355 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.388658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.388631 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.388933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.388666 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.388933 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.388683 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/09c7dec4-84e2-4e7d-ad60-9bca21493b43-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.390698 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.390659 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.390945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.390913 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.391282 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.391244 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "kube-api-access-rbr2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.391393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.391331 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.391440 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.391425 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.392032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.391973 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:10.392134 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.392063 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.392516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.392491 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.394078 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.394058 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:47:10.404383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.404363 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config" pod "09c7dec4-84e2-4e7d-ad60-9bca21493b43" (UID: "09c7dec4-84e2-4e7d-ad60-9bca21493b43"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.489494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489466 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489495 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489511 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbr2d\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-kube-api-access-rbr2d\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489544 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489561 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489576 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489589 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489603 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09c7dec4-84e2-4e7d-ad60-9bca21493b43-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489619 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.489852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.489634 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09c7dec4-84e2-4e7d-ad60-9bca21493b43-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.500536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.500521 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:47:10.555386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.555328 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" Oct 31 00:47:10.555386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.555355 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl" event={"ID":"98c62573-128b-46b3-b6e8-7b9b1f3b5bdb","Type":"ContainerDied","Data":"6fbb0d988ae1052482ef90d338e1eb20073c65f377cba78f92bf41ce4611c131"} Oct 31 00:47:10.555504 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.555405 5073 scope.go:122] "RemoveContainer" containerID="f2b9e4fbcafff6beec9b5eb87a3f0c8d9c2cc21e57b22b5ccd96997c7450b0f2" Oct 31 00:47:10.556809 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.556768 5073 generic.go:356] "Generic (PLEG): container finished" podID="7611cf43-05c1-4940-87a7-c66552ecdf99" containerID="3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6" exitCode=0 Oct 31 00:47:10.556904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.556825 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" event={"ID":"7611cf43-05c1-4940-87a7-c66552ecdf99","Type":"ContainerDied","Data":"3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6"} Oct 31 00:47:10.556904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.556837 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" Oct 31 00:47:10.556904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.556856 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl" event={"ID":"7611cf43-05c1-4940-87a7-c66552ecdf99","Type":"ContainerDied","Data":"3dbbc357394236e47a7a8b89c33405f439bb93ac1eb20fa59213584b478ee726"} Oct 31 00:47:10.559472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559451 5073 generic.go:356] "Generic (PLEG): container finished" podID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerID="cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" exitCode=0 Oct 31 00:47:10.559472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559470 5073 generic.go:356] "Generic (PLEG): container finished" podID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerID="e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" exitCode=0 Oct 31 00:47:10.559651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559479 5073 generic.go:356] "Generic (PLEG): container finished" podID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerID="0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" exitCode=0 Oct 31 00:47:10.559651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559513 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerDied","Data":"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e"} Oct 31 00:47:10.559651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559574 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerDied","Data":"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b"} Oct 31 00:47:10.559651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559515 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:10.559651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559602 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerDied","Data":"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61"} Oct 31 00:47:10.559651 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.559628 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"09c7dec4-84e2-4e7d-ad60-9bca21493b43","Type":"ContainerDied","Data":"3325c11e1ddca6128f2120088111e4599021e5538879df755779a298c4b9e1f8"} Oct 31 00:47:10.565333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.565314 5073 scope.go:122] "RemoveContainer" containerID="3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6" Oct 31 00:47:10.573502 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.573481 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl"] Oct 31 00:47:10.573737 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.573719 5073 scope.go:122] "RemoveContainer" containerID="3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6" Oct 31 00:47:10.574016 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:10.574000 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6\": container with ID starting with 3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6 not found: ID does not exist" containerID="3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6" Oct 31 00:47:10.574124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.574020 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6"} err="failed to get container status \"3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6\": rpc error: code = NotFound desc = could not find container \"3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6\": container with ID starting with 3516ca9edfa4bf4a62e480f6dddfbbee9708be74c2862aab48d03b81e4e752e6 not found: ID does not exist" Oct 31 00:47:10.574124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.574035 5073 scope.go:122] "RemoveContainer" containerID="cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" Oct 31 00:47:10.580713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.580691 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2rnwl"] Oct 31 00:47:10.581975 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.581961 5073 scope.go:122] "RemoveContainer" containerID="e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" Oct 31 00:47:10.590021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.589993 5073 scope.go:122] "RemoveContainer" containerID="0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" Oct 31 00:47:10.590220 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.590202 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" Oct 31 00:47:10.597889 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.597870 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl"] Oct 31 00:47:10.599059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.599042 5073 scope.go:122] "RemoveContainer" containerID="ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a" Oct 31 00:47:10.602726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.602702 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d7496d9fb-l4qsl"] Oct 31 00:47:10.607091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607076 5073 scope.go:122] "RemoveContainer" containerID="cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" Oct 31 00:47:10.607349 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:10.607330 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": container with ID starting with cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e not found: ID does not exist" containerID="cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" Oct 31 00:47:10.607422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607354 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e"} err="failed to get container status \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": rpc error: code = NotFound desc = could not find container \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": container with ID starting with cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e not found: ID does not exist" Oct 31 00:47:10.607422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607370 5073 scope.go:122] "RemoveContainer" containerID="e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" Oct 31 00:47:10.607629 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:10.607611 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": container with ID starting with e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b not found: ID does not exist" containerID="e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" Oct 31 00:47:10.607674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607634 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b"} err="failed to get container status \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": rpc error: code = NotFound desc = could not find container \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": container with ID starting with e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b not found: ID does not exist" Oct 31 00:47:10.607674 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607648 5073 scope.go:122] "RemoveContainer" containerID="0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" Oct 31 00:47:10.607884 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:10.607869 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": container with ID starting with 0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61 not found: ID does not exist" containerID="0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" Oct 31 00:47:10.607938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607888 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61"} err="failed to get container status \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": rpc error: code = NotFound desc = could not find container \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": container with ID starting with 0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61 not found: ID does not exist" Oct 31 00:47:10.607938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.607905 5073 scope.go:122] "RemoveContainer" containerID="ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a" Oct 31 00:47:10.608127 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:10.608112 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": container with ID starting with ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a not found: ID does not exist" containerID="ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a" Oct 31 00:47:10.608165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608130 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a"} err="failed to get container status \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": rpc error: code = NotFound desc = could not find container \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": container with ID starting with ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a not found: ID does not exist" Oct 31 00:47:10.608165 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608142 5073 scope.go:122] "RemoveContainer" containerID="cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" Oct 31 00:47:10.608362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608345 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e"} err="failed to get container status \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": rpc error: code = NotFound desc = could not find container \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": container with ID starting with cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e not found: ID does not exist" Oct 31 00:47:10.608408 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608362 5073 scope.go:122] "RemoveContainer" containerID="e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" Oct 31 00:47:10.608523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608508 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b"} err="failed to get container status \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": rpc error: code = NotFound desc = could not find container \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": container with ID starting with e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b not found: ID does not exist" Oct 31 00:47:10.608564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608523 5073 scope.go:122] "RemoveContainer" containerID="0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" Oct 31 00:47:10.608700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608686 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61"} err="failed to get container status \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": rpc error: code = NotFound desc = could not find container \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": container with ID starting with 0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61 not found: ID does not exist" Oct 31 00:47:10.608735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608700 5073 scope.go:122] "RemoveContainer" containerID="ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a" Oct 31 00:47:10.608926 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608912 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a"} err="failed to get container status \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": rpc error: code = NotFound desc = could not find container \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": container with ID starting with ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a not found: ID does not exist" Oct 31 00:47:10.608982 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.608926 5073 scope.go:122] "RemoveContainer" containerID="cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e" Oct 31 00:47:10.609126 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609109 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e"} err="failed to get container status \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": rpc error: code = NotFound desc = could not find container \"cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e\": container with ID starting with cb06e36bf868558fe21fc17a1925da85ab91432edde5680ba3415554dd81016e not found: ID does not exist" Oct 31 00:47:10.609168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609127 5073 scope.go:122] "RemoveContainer" containerID="e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b" Oct 31 00:47:10.609314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609300 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b"} err="failed to get container status \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": rpc error: code = NotFound desc = could not find container \"e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b\": container with ID starting with e941f18b59e3279b9603dc6e82890f7631e9c2de0e667d99c668f210548a5d6b not found: ID does not exist" Oct 31 00:47:10.609353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609314 5073 scope.go:122] "RemoveContainer" containerID="0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61" Oct 31 00:47:10.609525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609511 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61"} err="failed to get container status \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": rpc error: code = NotFound desc = could not find container \"0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61\": container with ID starting with 0af281c4a214050c7a655d52c48b1fc90a65c24a30b71fde3307fd6fecc8df61 not found: ID does not exist" Oct 31 00:47:10.609570 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609526 5073 scope.go:122] "RemoveContainer" containerID="ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a" Oct 31 00:47:10.609729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.609710 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a"} err="failed to get container status \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": rpc error: code = NotFound desc = could not find container \"ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a\": container with ID starting with ba61b978280f56aa233262a305df62f769edd951e5d998d8d35461502c603a3a not found: ID does not exist" Oct 31 00:47:10.616172 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.616152 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:47:10.621495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.621475 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:47:10.948631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:10.948550 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:12.060005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:12.059973 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" path="/var/lib/kubelet/pods/09c7dec4-84e2-4e7d-ad60-9bca21493b43/volumes" Oct 31 00:47:12.060497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:12.060481 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7611cf43-05c1-4940-87a7-c66552ecdf99" path="/var/lib/kubelet/pods/7611cf43-05c1-4940-87a7-c66552ecdf99/volumes" Oct 31 00:47:12.060769 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:12.060758 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" path="/var/lib/kubelet/pods/98c62573-128b-46b3-b6e8-7b9b1f3b5bdb/volumes" Oct 31 00:47:13.085063 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:13.085026 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:47:17.850074 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.849983 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7"] Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850420 5073 cpu_manager.go:414] "containerMap: removing container" podUID="98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" containerName="thanos-querier" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850439 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" containerName="thanos-querier" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850452 5073 cpu_manager.go:414] "containerMap: removing container" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="prometheus" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850458 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="prometheus" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850474 5073 cpu_manager.go:414] "containerMap: removing container" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="thanos-sidecar" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850481 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="thanos-sidecar" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850488 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7611cf43-05c1-4940-87a7-c66552ecdf99" containerName="ta-container" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850492 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7611cf43-05c1-4940-87a7-c66552ecdf99" containerName="ta-container" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850506 5073 cpu_manager.go:414] "containerMap: removing container" podUID="ffaa97a2-1a3b-4900-9502-d997146cd809" containerName="ta-container" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850511 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffaa97a2-1a3b-4900-9502-d997146cd809" containerName="ta-container" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850518 5073 cpu_manager.go:414] "containerMap: removing container" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="config-reloader" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850523 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="config-reloader" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850531 5073 cpu_manager.go:414] "containerMap: removing container" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="init-config-reloader" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850535 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="init-config-reloader" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850596 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffaa97a2-1a3b-4900-9502-d997146cd809" containerName="ta-container" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850604 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="config-reloader" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850614 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c62573-128b-46b3-b6e8-7b9b1f3b5bdb" containerName="thanos-querier" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850619 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="prometheus" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850629 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c7dec4-84e2-4e7d-ad60-9bca21493b43" containerName="thanos-sidecar" Oct 31 00:47:17.850648 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.850637 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7611cf43-05c1-4940-87a7-c66552ecdf99" containerName="ta-container" Oct 31 00:47:17.855993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.855973 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:17.860959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.860935 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:47:17.863265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.863235 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7"] Oct 31 00:47:17.952659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:17.952621 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7t7\" (UniqueName: \"kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7\" (UID: \"f1c0c59a-904c-4286-87db-67a379f29de0\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:18.053103 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.053064 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7t7\" (UniqueName: \"kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7\" (UID: \"f1c0c59a-904c-4286-87db-67a379f29de0\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:18.068997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.068966 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7t7\" (UniqueName: \"kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7\" (UID: \"f1c0c59a-904c-4286-87db-67a379f29de0\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:18.168924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.168838 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:18.219799 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.219745 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227"] Oct 31 00:47:18.226853 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.226828 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.243714 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.242505 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227"] Oct 31 00:47:18.251589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.250128 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:47:18.251589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.250268 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:47:18.251589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.250376 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-wxvbl\"" Oct 31 00:47:18.356475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.356435 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.356475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.356480 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmj9m\" (UniqueName: \"kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.356695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.356574 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.453534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.453454 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7"] Oct 31 00:47:18.456524 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:47:18.456495 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1c0c59a_904c_4286_87db_67a379f29de0.slice/crio-a26759318c24843305063b83d8b240ccbaab3f1fdac305a590a762edbb0ed096 WatchSource:0}: Error finding container a26759318c24843305063b83d8b240ccbaab3f1fdac305a590a762edbb0ed096: Status 404 returned error can't find the container with id a26759318c24843305063b83d8b240ccbaab3f1fdac305a590a762edbb0ed096 Oct 31 00:47:18.457490 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.457460 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.457609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.457514 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmj9m\" (UniqueName: \"kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.457609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.457576 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.458824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.458201 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.460606 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.460537 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.471155 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.471131 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmj9m\" (UniqueName: \"kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m\") pod \"data-science-collector-targetallocator-5867f5c45d-rf227\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.557564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.557534 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:18.621048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.621009 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" event={"ID":"f1c0c59a-904c-4286-87db-67a379f29de0","Type":"ContainerStarted","Data":"4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521"} Oct 31 00:47:18.621048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.621050 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" event={"ID":"f1c0c59a-904c-4286-87db-67a379f29de0","Type":"ContainerStarted","Data":"a26759318c24843305063b83d8b240ccbaab3f1fdac305a590a762edbb0ed096"} Oct 31 00:47:18.639289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.639235 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" podStartSLOduration=1.6392224130000002 podStartE2EDuration="1.639222413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:18.638465015 +0000 UTC m=+2922.699239734" watchObservedRunningTime="2026-10-31 00:47:18.639222413 +0000 UTC m=+2922.699997107" Oct 31 00:47:18.831859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.831827 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227"] Oct 31 00:47:18.834640 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:47:18.834613 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29755509_3d0d_4312_a4ac_a7737f2b194a.slice/crio-f6af87265885c2588b5e115da62eb6b59be9a420d650934cd71ed1f49f29fda0 WatchSource:0}: Error finding container f6af87265885c2588b5e115da62eb6b59be9a420d650934cd71ed1f49f29fda0: Status 404 returned error can't find the container with id f6af87265885c2588b5e115da62eb6b59be9a420d650934cd71ed1f49f29fda0 Oct 31 00:47:18.836834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.836773 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:47:18.836906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:18.836860 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:47:19.627441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:19.627399 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" event={"ID":"29755509-3d0d-4312-a4ac-a7737f2b194a","Type":"ContainerStarted","Data":"c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136"} Oct 31 00:47:19.627441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:19.627445 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" event={"ID":"29755509-3d0d-4312-a4ac-a7737f2b194a","Type":"ContainerStarted","Data":"f6af87265885c2588b5e115da62eb6b59be9a420d650934cd71ed1f49f29fda0"} Oct 31 00:47:19.627945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:19.627661 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:19.629425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:19.629404 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:19.648090 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:19.648033 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" podStartSLOduration=1.648020536 podStartE2EDuration="1.648020536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:19.647018472 +0000 UTC m=+2923.707793167" watchObservedRunningTime="2026-10-31 00:47:19.648020536 +0000 UTC m=+2923.708795233" Oct 31 00:47:20.948761 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:20.948723 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:24.259477 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.259447 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8"] Oct 31 00:47:24.262956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.262935 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:47:24.271284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.271260 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8"] Oct 31 00:47:24.410242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.410206 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8\" (UID: \"7f8e7464-4051-464a-bf86-497c71b5ea5e\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:47:24.511691 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.511611 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8\" (UID: \"7f8e7464-4051-464a-bf86-497c71b5ea5e\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:47:24.524400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.524373 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8\" (UID: \"7f8e7464-4051-464a-bf86-497c71b5ea5e\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:47:24.575830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.575808 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:47:24.852335 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:24.852195 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8"] Oct 31 00:47:24.854891 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:47:24.854863 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f8e7464_4051_464a_bf86_497c71b5ea5e.slice/crio-0fa7d280d5211ed9dbe00473d2f6e2ad903ef1d035063037046cadab0218d4ba WatchSource:0}: Error finding container 0fa7d280d5211ed9dbe00473d2f6e2ad903ef1d035063037046cadab0218d4ba: Status 404 returned error can't find the container with id 0fa7d280d5211ed9dbe00473d2f6e2ad903ef1d035063037046cadab0218d4ba Oct 31 00:47:25.227872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.227775 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:47:25.232664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.232641 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.236587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.236563 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:47:25.236732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.236629 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:47:25.236732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.236629 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:47:25.236732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.236633 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:47:25.236732 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.236655 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:47:25.237472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.237458 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:47:25.237540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.237522 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:47:25.237609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.237557 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:47:25.237609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.237569 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:47:25.237609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.237570 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dlnbv\"" Oct 31 00:47:25.247813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.247769 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:47:25.419100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419074 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419103 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419125 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419185 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419280 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419302 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419411 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwv9j\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419440 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419492 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419472 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419536 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419562 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.419772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.419583 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.519948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.519919 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.519983 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwv9j\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520001 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520023 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520048 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520063 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520091 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520115 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520137 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520329 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520371 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.520487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.520422 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.521168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.521077 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.521488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.521463 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.521904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.521829 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.523465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.523425 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.523539 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.523476 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.523539 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.523511 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.523715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.523690 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.523855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.523837 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.524082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.524060 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.524293 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.524279 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.524873 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.524857 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.533016 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.532996 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwv9j\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:25.656661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.656629 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" event={"ID":"7f8e7464-4051-464a-bf86-497c71b5ea5e","Type":"ContainerStarted","Data":"88cf0e4abf8a6ef87bd5f0ce5f0af0afc02aa2a253c6503607b1a3b48189f6c4"} Oct 31 00:47:25.656772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.656667 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" event={"ID":"7f8e7464-4051-464a-bf86-497c71b5ea5e","Type":"ContainerStarted","Data":"0fa7d280d5211ed9dbe00473d2f6e2ad903ef1d035063037046cadab0218d4ba"} Oct 31 00:47:25.674072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.674030 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" podStartSLOduration=1.674017279 podStartE2EDuration="1.674017279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:25.673344422 +0000 UTC m=+2929.734119116" watchObservedRunningTime="2026-10-31 00:47:25.674017279 +0000 UTC m=+2929.734791974" Oct 31 00:47:25.704465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.704431 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7"] Oct 31 00:47:25.704640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.704618 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" podUID="f1c0c59a-904c-4286-87db-67a379f29de0" containerName="thanos-querier" containerID="cri-o://4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521" gracePeriod=30 Oct 31 00:47:25.971320 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:25.971296 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:26.126094 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.126031 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7\" (UniqueName: \"kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7\") pod \"f1c0c59a-904c-4286-87db-67a379f29de0\" (UID: \"f1c0c59a-904c-4286-87db-67a379f29de0\") " Oct 31 00:47:26.128217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.128188 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7" pod "f1c0c59a-904c-4286-87db-67a379f29de0" (UID: "f1c0c59a-904c-4286-87db-67a379f29de0"). InnerVolumeSpecName "kube-api-access-kz7t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:26.227540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.227514 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz7t7\" (UniqueName: \"kubernetes.io/projected/f1c0c59a-904c-4286-87db-67a379f29de0-kube-api-access-kz7t7\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:26.662545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.662512 5073 generic.go:356] "Generic (PLEG): container finished" podID="f1c0c59a-904c-4286-87db-67a379f29de0" containerID="4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521" exitCode=0 Oct 31 00:47:26.663000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.662560 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" Oct 31 00:47:26.663000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.662599 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" event={"ID":"f1c0c59a-904c-4286-87db-67a379f29de0","Type":"ContainerDied","Data":"4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521"} Oct 31 00:47:26.663000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.662654 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7" event={"ID":"f1c0c59a-904c-4286-87db-67a379f29de0","Type":"ContainerDied","Data":"a26759318c24843305063b83d8b240ccbaab3f1fdac305a590a762edbb0ed096"} Oct 31 00:47:26.663000 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.662682 5073 scope.go:122] "RemoveContainer" containerID="4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521" Oct 31 00:47:26.671658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.671641 5073 scope.go:122] "RemoveContainer" containerID="4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521" Oct 31 00:47:26.672041 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:26.672019 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521\": container with ID starting with 4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521 not found: ID does not exist" containerID="4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521" Oct 31 00:47:26.672125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.672052 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521"} err="failed to get container status \"4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521\": rpc error: code = NotFound desc = could not find container \"4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521\": container with ID starting with 4aa3dbc917d4cf867c679291868d1225a2728e3426a5356c1ba8bb7a11a9c521 not found: ID does not exist" Oct 31 00:47:26.684107 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.684086 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7"] Oct 31 00:47:26.690280 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:26.690259 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-prtv7"] Oct 31 00:47:28.057898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:28.057868 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c0c59a-904c-4286-87db-67a379f29de0" path="/var/lib/kubelet/pods/f1c0c59a-904c-4286-87db-67a379f29de0/volumes" Oct 31 00:47:30.948862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:30.948828 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:34.796915 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:34.796883 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:34.803977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:34.803947 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:34.897776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:34.897745 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:34.897919 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:34.897888 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:34.901377 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:34.901355 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:34.992428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:34.992401 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:35.007988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:35.007961 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:35.152408 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:35.152354 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:35.442316 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:35.442161 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:47:35.444573 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:47:35.444546 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfac6b78_d506_4fab_8ce7_e12b884ec080.slice/crio-ec62b22e6bbecab0dd09f9111b726edbfe0532a7f42f82e9d5a222a7acb114f2 WatchSource:0}: Error finding container ec62b22e6bbecab0dd09f9111b726edbfe0532a7f42f82e9d5a222a7acb114f2: Status 404 returned error can't find the container with id ec62b22e6bbecab0dd09f9111b726edbfe0532a7f42f82e9d5a222a7acb114f2 Oct 31 00:47:35.704927 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:35.704849 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerStarted","Data":"ec62b22e6bbecab0dd09f9111b726edbfe0532a7f42f82e9d5a222a7acb114f2"} Oct 31 00:47:36.710862 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:36.710826 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerStarted","Data":"3eb487a17fa59b10e06e5dd75e7acddf8aa2168458f3b676d0c09cdb156563fb"} Oct 31 00:47:39.725667 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:39.725630 5073 generic.go:356] "Generic (PLEG): container finished" podID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerID="3eb487a17fa59b10e06e5dd75e7acddf8aa2168458f3b676d0c09cdb156563fb" exitCode=0 Oct 31 00:47:39.726148 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:39.725670 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerDied","Data":"3eb487a17fa59b10e06e5dd75e7acddf8aa2168458f3b676d0c09cdb156563fb"} Oct 31 00:47:40.733160 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:40.733125 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerStarted","Data":"2b8a3ff3467646b36e15e59ab9014162ca658a5f355902dd004ae7c1c6da05ba"} Oct 31 00:47:40.733586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:40.733193 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerStarted","Data":"3adaa054112a48b68dddb4821ea00466cffa3ca6d71d55e4377303178785af43"} Oct 31 00:47:40.733586 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:40.733218 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerStarted","Data":"3ce4da8dea7db437320996f5401ad26c7548a8a46cbfb245ab8c09dd73d04afb"} Oct 31 00:47:40.767089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:40.767042 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=12.381677006 podStartE2EDuration="15.767028332s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385351326s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:40.765199253 +0000 UTC m=+2944.825973949" watchObservedRunningTime="2026-10-31 00:47:40.767028332 +0000 UTC m=+2944.827803026" Oct 31 00:47:40.948656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:40.948617 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:47:43.087775 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:43.087736 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:47:45.153025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:45.152988 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:47.484753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.484712 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9"] Oct 31 00:47:47.485311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.485092 5073 cpu_manager.go:414] "containerMap: removing container" podUID="f1c0c59a-904c-4286-87db-67a379f29de0" containerName="thanos-querier" Oct 31 00:47:47.485311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.485104 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c0c59a-904c-4286-87db-67a379f29de0" containerName="thanos-querier" Oct 31 00:47:47.485311 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.485179 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c0c59a-904c-4286-87db-67a379f29de0" containerName="thanos-querier" Oct 31 00:47:47.488309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.488284 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.498727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.498698 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9"] Oct 31 00:47:47.600162 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.600126 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.600348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.600181 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.600348 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.600291 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjddk\" (UniqueName: \"kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.701546 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.701508 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjddk\" (UniqueName: \"kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.702115 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.701629 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.702359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.702337 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.702776 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.702747 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.710763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.710740 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.714313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.714295 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjddk\" (UniqueName: \"kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk\") pod \"data-science-collector-targetallocator-7db79f8df9-wsbz9\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:47.804296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:47.804212 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:48.080203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.080177 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9"] Oct 31 00:47:48.082906 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:47:48.082879 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd52132f9_49c3_4ff5_a9f6_f979d292917f.slice/crio-806d85ad2ae1b5680f7ea89521ea7cea2488cc81bfebeeaadeeb2e45eeff1159 WatchSource:0}: Error finding container 806d85ad2ae1b5680f7ea89521ea7cea2488cc81bfebeeaadeeb2e45eeff1159: Status 404 returned error can't find the container with id 806d85ad2ae1b5680f7ea89521ea7cea2488cc81bfebeeaadeeb2e45eeff1159 Oct 31 00:47:48.085484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.085442 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:47:48.085556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.085526 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:47:48.771601 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.771561 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" event={"ID":"d52132f9-49c3-4ff5-a9f6-f979d292917f","Type":"ContainerStarted","Data":"afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed"} Oct 31 00:47:48.772069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.771613 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" event={"ID":"d52132f9-49c3-4ff5-a9f6-f979d292917f","Type":"ContainerStarted","Data":"806d85ad2ae1b5680f7ea89521ea7cea2488cc81bfebeeaadeeb2e45eeff1159"} Oct 31 00:47:48.772069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.771635 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:48.773232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.773212 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:47:48.796640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.796556 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" podStartSLOduration=1.796542288 podStartE2EDuration="1.796542288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:48.794850373 +0000 UTC m=+2952.855625069" watchObservedRunningTime="2026-10-31 00:47:48.796542288 +0000 UTC m=+2952.857316982" Oct 31 00:47:48.848129 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.848095 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227"] Oct 31 00:47:48.848308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:48.848286 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" podUID="29755509-3d0d-4312-a4ac-a7737f2b194a" containerName="ta-container" containerID="cri-o://c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136" gracePeriod=30 Oct 31 00:47:49.111700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.111672 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:49.215666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.215634 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal\" (UniqueName: \"kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal\") pod \"29755509-3d0d-4312-a4ac-a7737f2b194a\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " Oct 31 00:47:49.215836 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.215689 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m\" (UniqueName: \"kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m\") pod \"29755509-3d0d-4312-a4ac-a7737f2b194a\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " Oct 31 00:47:49.215897 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.215845 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert\") pod \"29755509-3d0d-4312-a4ac-a7737f2b194a\" (UID: \"29755509-3d0d-4312-a4ac-a7737f2b194a\") " Oct 31 00:47:49.216034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.216016 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal" pod "29755509-3d0d-4312-a4ac-a7737f2b194a" (UID: "29755509-3d0d-4312-a4ac-a7737f2b194a"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:49.216179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.216159 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/29755509-3d0d-4312-a4ac-a7737f2b194a-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.217824 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.217803 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m" pod "29755509-3d0d-4312-a4ac-a7737f2b194a" (UID: "29755509-3d0d-4312-a4ac-a7737f2b194a"). InnerVolumeSpecName "kube-api-access-qmj9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:49.217924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.217888 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert" pod "29755509-3d0d-4312-a4ac-a7737f2b194a" (UID: "29755509-3d0d-4312-a4ac-a7737f2b194a"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:49.317475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.317445 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmj9m\" (UniqueName: \"kubernetes.io/projected/29755509-3d0d-4312-a4ac-a7737f2b194a-kube-api-access-qmj9m\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.317475 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.317474 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/29755509-3d0d-4312-a4ac-a7737f2b194a-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.777773 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.777743 5073 generic.go:356] "Generic (PLEG): container finished" podID="29755509-3d0d-4312-a4ac-a7737f2b194a" containerID="c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136" exitCode=0 Oct 31 00:47:49.778286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.777796 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" Oct 31 00:47:49.778286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.777826 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" event={"ID":"29755509-3d0d-4312-a4ac-a7737f2b194a","Type":"ContainerDied","Data":"c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136"} Oct 31 00:47:49.778286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.777871 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227" event={"ID":"29755509-3d0d-4312-a4ac-a7737f2b194a","Type":"ContainerDied","Data":"f6af87265885c2588b5e115da62eb6b59be9a420d650934cd71ed1f49f29fda0"} Oct 31 00:47:49.778286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.777888 5073 scope.go:122] "RemoveContainer" containerID="c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136" Oct 31 00:47:49.787142 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.787126 5073 scope.go:122] "RemoveContainer" containerID="c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136" Oct 31 00:47:49.787442 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:47:49.787423 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136\": container with ID starting with c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136 not found: ID does not exist" containerID="c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136" Oct 31 00:47:49.787499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.787451 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136"} err="failed to get container status \"c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136\": rpc error: code = NotFound desc = could not find container \"c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136\": container with ID starting with c95b21b0fa77ffad7294bc3dc2c3e83d5f4f141663445b0344c3aaf06b73c136 not found: ID does not exist" Oct 31 00:47:49.800032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.800007 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227"] Oct 31 00:47:49.808944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:49.808918 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5867f5c45d-rf227"] Oct 31 00:47:50.058840 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:50.058753 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29755509-3d0d-4312-a4ac-a7737f2b194a" path="/var/lib/kubelet/pods/29755509-3d0d-4312-a4ac-a7737f2b194a/volumes" Oct 31 00:47:50.152793 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:50.152769 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:50.158368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:50.158348 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:50.787243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:50.787213 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:47:50.949523 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:47:50.949491 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:00.948549 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:00.948503 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:04.664046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.664016 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g"] Oct 31 00:48:04.664449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.664347 5073 cpu_manager.go:414] "containerMap: removing container" podUID="29755509-3d0d-4312-a4ac-a7737f2b194a" containerName="ta-container" Oct 31 00:48:04.664449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.664358 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29755509-3d0d-4312-a4ac-a7737f2b194a" containerName="ta-container" Oct 31 00:48:04.664449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.664442 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="29755509-3d0d-4312-a4ac-a7737f2b194a" containerName="ta-container" Oct 31 00:48:04.667613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.667594 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.676978 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.676949 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g"] Oct 31 00:48:04.840567 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.840536 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.840771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.840588 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.840771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.840676 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6wc\" (UniqueName: \"kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.941232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.941169 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.941232 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.941205 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6wc\" (UniqueName: \"kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.941408 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.941272 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.941916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.941897 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.943671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.943651 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.954263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.954227 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6wc\" (UniqueName: \"kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc\") pod \"data-science-collector-targetallocator-f69669c96-2lg8g\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:04.980117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:04.980096 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:05.255738 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.255639 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g"] Oct 31 00:48:05.258661 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:05.258635 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fdb492e_4ec0_480d_beb8_38f4e0c3a4fb.slice/crio-82e8d0d89bd1f56e212d1a417fe16f71f387244d65ecea3e571a1fd69bc9ddd3 WatchSource:0}: Error finding container 82e8d0d89bd1f56e212d1a417fe16f71f387244d65ecea3e571a1fd69bc9ddd3: Status 404 returned error can't find the container with id 82e8d0d89bd1f56e212d1a417fe16f71f387244d65ecea3e571a1fd69bc9ddd3 Oct 31 00:48:05.261003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.260967 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:05.261117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.261037 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:05.855432 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.855393 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" event={"ID":"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb","Type":"ContainerStarted","Data":"f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390"} Oct 31 00:48:05.855866 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.855450 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:05.855866 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.855467 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" event={"ID":"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb","Type":"ContainerStarted","Data":"82e8d0d89bd1f56e212d1a417fe16f71f387244d65ecea3e571a1fd69bc9ddd3"} Oct 31 00:48:05.857100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.857076 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:05.876816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.876748 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" podStartSLOduration=1.876735185 podStartE2EDuration="1.876735185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:05.875040343 +0000 UTC m=+2969.935815038" watchObservedRunningTime="2026-10-31 00:48:05.876735185 +0000 UTC m=+2969.937509880" Oct 31 00:48:05.926085 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.926055 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9"] Oct 31 00:48:05.926241 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:05.926220 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" podUID="d52132f9-49c3-4ff5-a9f6-f979d292917f" containerName="ta-container" containerID="cri-o://afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed" gracePeriod=30 Oct 31 00:48:06.201190 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.201166 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:48:06.354820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.354769 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal\") pod \"d52132f9-49c3-4ff5-a9f6-f979d292917f\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " Oct 31 00:48:06.355004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.354857 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert\") pod \"d52132f9-49c3-4ff5-a9f6-f979d292917f\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " Oct 31 00:48:06.355004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.354930 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk\" (UniqueName: \"kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk\") pod \"d52132f9-49c3-4ff5-a9f6-f979d292917f\" (UID: \"d52132f9-49c3-4ff5-a9f6-f979d292917f\") " Oct 31 00:48:06.355210 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.355172 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal" pod "d52132f9-49c3-4ff5-a9f6-f979d292917f" (UID: "d52132f9-49c3-4ff5-a9f6-f979d292917f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:06.357250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.357220 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert" pod "d52132f9-49c3-4ff5-a9f6-f979d292917f" (UID: "d52132f9-49c3-4ff5-a9f6-f979d292917f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:06.357250 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.357224 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk" pod "d52132f9-49c3-4ff5-a9f6-f979d292917f" (UID: "d52132f9-49c3-4ff5-a9f6-f979d292917f"). InnerVolumeSpecName "kube-api-access-kjddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:06.455685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.455616 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjddk\" (UniqueName: \"kubernetes.io/projected/d52132f9-49c3-4ff5-a9f6-f979d292917f-kube-api-access-kjddk\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.455685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.455644 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d52132f9-49c3-4ff5-a9f6-f979d292917f-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.455685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.455655 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d52132f9-49c3-4ff5-a9f6-f979d292917f-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.861458 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.861420 5073 generic.go:356] "Generic (PLEG): container finished" podID="d52132f9-49c3-4ff5-a9f6-f979d292917f" containerID="afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed" exitCode=0 Oct 31 00:48:06.861936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.861466 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" Oct 31 00:48:06.861936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.861504 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" event={"ID":"d52132f9-49c3-4ff5-a9f6-f979d292917f","Type":"ContainerDied","Data":"afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed"} Oct 31 00:48:06.861936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.861551 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9" event={"ID":"d52132f9-49c3-4ff5-a9f6-f979d292917f","Type":"ContainerDied","Data":"806d85ad2ae1b5680f7ea89521ea7cea2488cc81bfebeeaadeeb2e45eeff1159"} Oct 31 00:48:06.861936 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.861572 5073 scope.go:122] "RemoveContainer" containerID="afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed" Oct 31 00:48:06.870727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.870706 5073 scope.go:122] "RemoveContainer" containerID="afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed" Oct 31 00:48:06.871033 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:48:06.871011 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed\": container with ID starting with afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed not found: ID does not exist" containerID="afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed" Oct 31 00:48:06.871106 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.871042 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed"} err="failed to get container status \"afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed\": rpc error: code = NotFound desc = could not find container \"afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed\": container with ID starting with afdd7b4f7ba11affe93a692d442c1e46c72e451fcbea73319aa3943cf866c7ed not found: ID does not exist" Oct 31 00:48:06.885516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.885495 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9"] Oct 31 00:48:06.890605 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:06.890578 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7db79f8df9-wsbz9"] Oct 31 00:48:08.058421 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:08.058385 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52132f9-49c3-4ff5-a9f6-f979d292917f" path="/var/lib/kubelet/pods/d52132f9-49c3-4ff5-a9f6-f979d292917f/volumes" Oct 31 00:48:10.425359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.425323 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8"] Oct 31 00:48:10.426207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.426167 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" podUID="7f8e7464-4051-464a-bf86-497c71b5ea5e" containerName="thanos-querier" containerID="cri-o://88cf0e4abf8a6ef87bd5f0ce5f0af0afc02aa2a253c6503607b1a3b48189f6c4" gracePeriod=30 Oct 31 00:48:10.526960 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.526930 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:48:10.527263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.527232 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="prometheus" containerID="cri-o://3ce4da8dea7db437320996f5401ad26c7548a8a46cbfb245ab8c09dd73d04afb" gracePeriod=600 Oct 31 00:48:10.527345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.527262 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="config-reloader" containerID="cri-o://3adaa054112a48b68dddb4821ea00466cffa3ca6d71d55e4377303178785af43" gracePeriod=600 Oct 31 00:48:10.527345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.527286 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="thanos-sidecar" containerID="cri-o://2b8a3ff3467646b36e15e59ab9014162ca658a5f355902dd004ae7c1c6da05ba" gracePeriod=600 Oct 31 00:48:10.762375 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.761825 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g"] Oct 31 00:48:10.762375 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.762025 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" podUID="6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" containerName="ta-container" containerID="cri-o://f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390" gracePeriod=30 Oct 31 00:48:10.787431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.787186 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="prometheus" probeResult="failure" output="Get \"https://10.132.0.204:9090/-/ready\": dial tcp 10.132.0.204:9090: connect: connection refused" Oct 31 00:48:10.888820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.887364 5073 generic.go:356] "Generic (PLEG): container finished" podID="7f8e7464-4051-464a-bf86-497c71b5ea5e" containerID="88cf0e4abf8a6ef87bd5f0ce5f0af0afc02aa2a253c6503607b1a3b48189f6c4" exitCode=0 Oct 31 00:48:10.888820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.887401 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" event={"ID":"7f8e7464-4051-464a-bf86-497c71b5ea5e","Type":"ContainerDied","Data":"88cf0e4abf8a6ef87bd5f0ce5f0af0afc02aa2a253c6503607b1a3b48189f6c4"} Oct 31 00:48:10.888820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.887465 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" event={"ID":"7f8e7464-4051-464a-bf86-497c71b5ea5e","Type":"ContainerDied","Data":"0fa7d280d5211ed9dbe00473d2f6e2ad903ef1d035063037046cadab0218d4ba"} Oct 31 00:48:10.888820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.887483 5073 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa7d280d5211ed9dbe00473d2f6e2ad903ef1d035063037046cadab0218d4ba" Oct 31 00:48:10.890973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.890949 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:48:10.891217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.891185 5073 generic.go:356] "Generic (PLEG): container finished" podID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerID="2b8a3ff3467646b36e15e59ab9014162ca658a5f355902dd004ae7c1c6da05ba" exitCode=0 Oct 31 00:48:10.891217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.891216 5073 generic.go:356] "Generic (PLEG): container finished" podID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerID="3adaa054112a48b68dddb4821ea00466cffa3ca6d71d55e4377303178785af43" exitCode=0 Oct 31 00:48:10.891396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.891225 5073 generic.go:356] "Generic (PLEG): container finished" podID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerID="3ce4da8dea7db437320996f5401ad26c7548a8a46cbfb245ab8c09dd73d04afb" exitCode=0 Oct 31 00:48:10.891396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.891280 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerDied","Data":"2b8a3ff3467646b36e15e59ab9014162ca658a5f355902dd004ae7c1c6da05ba"} Oct 31 00:48:10.891396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.891338 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerDied","Data":"3adaa054112a48b68dddb4821ea00466cffa3ca6d71d55e4377303178785af43"} Oct 31 00:48:10.891396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.891376 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerDied","Data":"3ce4da8dea7db437320996f5401ad26c7548a8a46cbfb245ab8c09dd73d04afb"} Oct 31 00:48:10.931710 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.931684 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:10.951861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.950739 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:10.993842 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.993812 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d\") pod \"7f8e7464-4051-464a-bf86-497c71b5ea5e\" (UID: \"7f8e7464-4051-464a-bf86-497c71b5ea5e\") " Oct 31 00:48:10.996621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:10.996588 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d" pod "7f8e7464-4051-464a-bf86-497c71b5ea5e" (UID: "7f8e7464-4051-464a-bf86-497c71b5ea5e"). InnerVolumeSpecName "kube-api-access-9sn4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.085487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.085460 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:11.094777 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.094753 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.094964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.094810 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.094964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.094845 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.094964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.094865 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.094964 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.094885 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095188 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:11.095286 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095262 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:11.095344 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095332 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095349 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:11.095402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095385 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095420 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095460 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095506 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095547 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095677 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095641 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.095804 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.095701 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets\") pod \"cfac6b78-d506-4fab-8ce7-e12b884ec080\" (UID: \"cfac6b78-d506-4fab-8ce7-e12b884ec080\") " Oct 31 00:48:11.096291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.096206 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sn4d\" (UniqueName: \"kubernetes.io/projected/7f8e7464-4051-464a-bf86-497c71b5ea5e-kube-api-access-9sn4d\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.096291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.096232 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.096291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.096250 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.096291 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.096266 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cfac6b78-d506-4fab-8ce7-e12b884ec080-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.098280 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.098243 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.098953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.098917 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.099865 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.099832 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "kube-api-access-qwv9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.100739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.100713 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:11.101633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.101608 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.102641 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.102621 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.103177 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.103151 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.103386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.103348 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.105207 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.105185 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:48:11.120327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.120299 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config" pod "cfac6b78-d506-4fab-8ce7-e12b884ec080" (UID: "cfac6b78-d506-4fab-8ce7-e12b884ec080"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.197519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.197489 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc\" (UniqueName: \"kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc\") pod \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " Oct 31 00:48:11.197687 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.197576 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal\" (UniqueName: \"kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal\") pod \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " Oct 31 00:48:11.197771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.197702 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert\") pod \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\" (UID: \"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb\") " Oct 31 00:48:11.198009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.197965 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal" pod "6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" (UID: "6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:11.198009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.197995 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198014 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198029 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwv9j\" (UniqueName: \"kubernetes.io/projected/cfac6b78-d506-4fab-8ce7-e12b884ec080-kube-api-access-qwv9j\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198043 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198056 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198071 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198089 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198104 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cfac6b78-d506-4fab-8ce7-e12b884ec080-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198120 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/cfac6b78-d506-4fab-8ce7-e12b884ec080-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.198368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.198147 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:48:11.200609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.200586 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc" pod "6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" (UID: "6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb"). InnerVolumeSpecName "kube-api-access-sd6wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.200726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.200680 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert" pod "6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" (UID: "6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.216799 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.216762 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:48:11.299393 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.299364 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" Oct 31 00:48:11.299550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.299412 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.299550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.299426 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6wc\" (UniqueName: \"kubernetes.io/projected/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-kube-api-access-sd6wc\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.299550 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.299441 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.898706 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.898669 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"cfac6b78-d506-4fab-8ce7-e12b884ec080","Type":"ContainerDied","Data":"ec62b22e6bbecab0dd09f9111b726edbfe0532a7f42f82e9d5a222a7acb114f2"} Oct 31 00:48:11.899143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.898772 5073 scope.go:122] "RemoveContainer" containerID="2b8a3ff3467646b36e15e59ab9014162ca658a5f355902dd004ae7c1c6da05ba" Oct 31 00:48:11.899143 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.898686 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:11.900544 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.900518 5073 generic.go:356] "Generic (PLEG): container finished" podID="6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" containerID="f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390" exitCode=0 Oct 31 00:48:11.900711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.900577 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" Oct 31 00:48:11.900711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.900600 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" event={"ID":"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb","Type":"ContainerDied","Data":"f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390"} Oct 31 00:48:11.900711 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.900652 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g" event={"ID":"6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb","Type":"ContainerDied","Data":"82e8d0d89bd1f56e212d1a417fe16f71f387244d65ecea3e571a1fd69bc9ddd3"} Oct 31 00:48:11.900938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.900808 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8" Oct 31 00:48:11.908396 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.908374 5073 scope.go:122] "RemoveContainer" containerID="3adaa054112a48b68dddb4821ea00466cffa3ca6d71d55e4377303178785af43" Oct 31 00:48:11.917501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.917481 5073 scope.go:122] "RemoveContainer" containerID="3ce4da8dea7db437320996f5401ad26c7548a8a46cbfb245ab8c09dd73d04afb" Oct 31 00:48:11.925297 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.925279 5073 scope.go:122] "RemoveContainer" containerID="3eb487a17fa59b10e06e5dd75e7acddf8aa2168458f3b676d0c09cdb156563fb" Oct 31 00:48:11.928033 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.928012 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g"] Oct 31 00:48:11.932859 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.932838 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-f69669c96-2lg8g"] Oct 31 00:48:11.933670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.933656 5073 scope.go:122] "RemoveContainer" containerID="f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390" Oct 31 00:48:11.940932 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.940916 5073 scope.go:122] "RemoveContainer" containerID="f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390" Oct 31 00:48:11.941200 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:48:11.941181 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390\": container with ID starting with f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390 not found: ID does not exist" containerID="f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390" Oct 31 00:48:11.941256 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.941208 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390"} err="failed to get container status \"f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390\": rpc error: code = NotFound desc = could not find container \"f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390\": container with ID starting with f8422e3e5c0cb2efdcb6a292d9514dd5575899f62cdca8fef609b431e6969390 not found: ID does not exist" Oct 31 00:48:11.947274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.947252 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:48:11.952855 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.952833 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:48:11.965017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.964996 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8"] Oct 31 00:48:11.970019 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:11.970001 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-5sxs8"] Oct 31 00:48:12.058242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:12.058218 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" path="/var/lib/kubelet/pods/6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb/volumes" Oct 31 00:48:12.058572 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:12.058559 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8e7464-4051-464a-bf86-497c71b5ea5e" path="/var/lib/kubelet/pods/7f8e7464-4051-464a-bf86-497c71b5ea5e/volumes" Oct 31 00:48:12.059003 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:12.058989 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" path="/var/lib/kubelet/pods/cfac6b78-d506-4fab-8ce7-e12b884ec080/volumes" Oct 31 00:48:13.085484 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:48:13.085454 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:48:18.613922 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.613891 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls"] Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614202 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d52132f9-49c3-4ff5-a9f6-f979d292917f" containerName="ta-container" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614212 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52132f9-49c3-4ff5-a9f6-f979d292917f" containerName="ta-container" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614226 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="init-config-reloader" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614232 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="init-config-reloader" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614251 5073 cpu_manager.go:414] "containerMap: removing container" podUID="6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" containerName="ta-container" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614258 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" containerName="ta-container" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614278 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7f8e7464-4051-464a-bf86-497c71b5ea5e" containerName="thanos-querier" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614283 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8e7464-4051-464a-bf86-497c71b5ea5e" containerName="thanos-querier" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614290 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="config-reloader" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614294 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="config-reloader" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614300 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="thanos-sidecar" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614304 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="thanos-sidecar" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614313 5073 cpu_manager.go:414] "containerMap: removing container" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="prometheus" Oct 31 00:48:18.614330 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614318 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="prometheus" Oct 31 00:48:18.615004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614368 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8e7464-4051-464a-bf86-497c71b5ea5e" containerName="thanos-querier" Oct 31 00:48:18.615004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614376 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="config-reloader" Oct 31 00:48:18.615004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614383 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="thanos-sidecar" Oct 31 00:48:18.615004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614392 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfac6b78-d506-4fab-8ce7-e12b884ec080" containerName="prometheus" Oct 31 00:48:18.615004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614401 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52132f9-49c3-4ff5-a9f6-f979d292917f" containerName="ta-container" Oct 31 00:48:18.615004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.614411 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdb492e-4ec0-480d-beb8-38f4e0c3a4fb" containerName="ta-container" Oct 31 00:48:18.619026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.619006 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:18.629289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.629256 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:48:18.629981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.629958 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls"] Oct 31 00:48:18.752755 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.752728 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56x8\" (UniqueName: \"kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls\" (UID: \"d397f027-2786-4b6c-9876-83ef098f31f6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:18.853175 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.853151 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56x8\" (UniqueName: \"kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls\" (UID: \"d397f027-2786-4b6c-9876-83ef098f31f6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:18.866203 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.866151 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56x8\" (UniqueName: \"kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls\" (UID: \"d397f027-2786-4b6c-9876-83ef098f31f6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:18.930580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:18.930551 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:19.006998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.006962 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb"] Oct 31 00:48:19.019857 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.019624 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.021383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.021354 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb"] Oct 31 00:48:19.039717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.038530 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-z85lc\"" Oct 31 00:48:19.039717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.038654 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:48:19.039717 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.038731 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:48:19.155034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.154957 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.155178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.155059 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.155178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.155116 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvqq\" (UniqueName: \"kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.222105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.222077 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls"] Oct 31 00:48:19.225067 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:19.225042 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd397f027_2786_4b6c_9876_83ef098f31f6.slice/crio-3f674ca7d37f2ba52a9b9bcd049b28bc5804589888725f50a226259e90f22b6b WatchSource:0}: Error finding container 3f674ca7d37f2ba52a9b9bcd049b28bc5804589888725f50a226259e90f22b6b: Status 404 returned error can't find the container with id 3f674ca7d37f2ba52a9b9bcd049b28bc5804589888725f50a226259e90f22b6b Oct 31 00:48:19.256702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.256678 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.256814 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.256739 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.256814 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.256803 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvqq\" (UniqueName: \"kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.257354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.257330 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.259399 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.259351 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.270534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.270512 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvqq\" (UniqueName: \"kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq\") pod \"data-science-collector-targetallocator-bf55598c4-4cjlb\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.336734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.336711 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.612401 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.612273 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb"] Oct 31 00:48:19.615264 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:19.615233 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd07221d_a0fa_48d6_b698_174f08a3bb5c.slice/crio-652651fdb871f845af2f14fa3a4de5ed658877286a0e83cce964b6da3db86181 WatchSource:0}: Error finding container 652651fdb871f845af2f14fa3a4de5ed658877286a0e83cce964b6da3db86181: Status 404 returned error can't find the container with id 652651fdb871f845af2f14fa3a4de5ed658877286a0e83cce964b6da3db86181 Oct 31 00:48:19.617441 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.617408 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:19.617529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.617465 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:19.943501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.943423 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" event={"ID":"fd07221d-a0fa-48d6-b698-174f08a3bb5c","Type":"ContainerStarted","Data":"4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274"} Oct 31 00:48:19.943501 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.943465 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" event={"ID":"fd07221d-a0fa-48d6-b698-174f08a3bb5c","Type":"ContainerStarted","Data":"652651fdb871f845af2f14fa3a4de5ed658877286a0e83cce964b6da3db86181"} Oct 31 00:48:19.943705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.943600 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.945253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.945224 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" event={"ID":"d397f027-2786-4b6c-9876-83ef098f31f6","Type":"ContainerStarted","Data":"a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba"} Oct 31 00:48:19.945376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.945262 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" event={"ID":"d397f027-2786-4b6c-9876-83ef098f31f6","Type":"ContainerStarted","Data":"3f674ca7d37f2ba52a9b9bcd049b28bc5804589888725f50a226259e90f22b6b"} Oct 31 00:48:19.945376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.945339 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:19.964080 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.964038 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" podStartSLOduration=1.964025527 podStartE2EDuration="1.964025527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:19.962818931 +0000 UTC m=+2984.023593629" watchObservedRunningTime="2026-10-31 00:48:19.964025527 +0000 UTC m=+2984.024800222" Oct 31 00:48:19.980541 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:19.980501 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" podStartSLOduration=1.980489628 podStartE2EDuration="1.980489628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:19.980153411 +0000 UTC m=+2984.040928108" watchObservedRunningTime="2026-10-31 00:48:19.980489628 +0000 UTC m=+2984.041264319" Oct 31 00:48:20.949015 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:20.948980 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:24.377739 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.376080 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944"] Oct 31 00:48:24.380569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.380543 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:48:24.389760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.389736 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944"] Oct 31 00:48:24.396273 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.396245 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldzwr\" (UniqueName: \"kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-l6944\" (UID: \"1135c40a-4e9e-45e5-9aef-5e4373437998\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:48:24.497031 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.496998 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldzwr\" (UniqueName: \"kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-l6944\" (UID: \"1135c40a-4e9e-45e5-9aef-5e4373437998\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:48:24.509652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.509628 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldzwr\" (UniqueName: \"kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-l6944\" (UID: \"1135c40a-4e9e-45e5-9aef-5e4373437998\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:48:24.704989 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.704923 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:48:24.986487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:24.986455 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944"] Oct 31 00:48:24.989568 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:24.989536 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1135c40a_4e9e_45e5_9aef_5e4373437998.slice/crio-32ec9b59dfcc92394ff7f93ac92dcb29ae5886bcbc807ca731540cf477170077 WatchSource:0}: Error finding container 32ec9b59dfcc92394ff7f93ac92dcb29ae5886bcbc807ca731540cf477170077: Status 404 returned error can't find the container with id 32ec9b59dfcc92394ff7f93ac92dcb29ae5886bcbc807ca731540cf477170077 Oct 31 00:48:25.347702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.347630 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:48:25.351455 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.351437 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.355809 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.355773 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:48:25.355809 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.355797 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:48:25.356350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.356015 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:48:25.356350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.356063 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:48:25.356350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.356069 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:48:25.356350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.356068 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:48:25.356350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.356065 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:48:25.356988 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.356971 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:48:25.357253 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.357239 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-k9mwh\"" Oct 31 00:48:25.357313 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.357296 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:48:25.368944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.368922 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:48:25.505039 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505009 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505045 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505076 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505188 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505243 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505306 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505355 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505388 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505408 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjknm\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505501 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505522 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.505827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.505591 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.606962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.606888 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.606962 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.606925 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.606962 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.606989 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607013 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607042 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607070 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607159 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607127 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607160 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607197 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607238 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.607493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.607264 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjknm\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.608173 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.608149 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.608538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.608509 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.610096 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.610068 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.610364 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.610242 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.610495 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.610474 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.610632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.610609 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.610762 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.610741 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.610997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.610972 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.611097 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.611083 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.611138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.611117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.611399 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.611379 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.619778 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.619754 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjknm\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:25.973622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.973532 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" event={"ID":"1135c40a-4e9e-45e5-9aef-5e4373437998","Type":"ContainerStarted","Data":"0e62d7fd77b70d0b595d9238934610bd2b4386a898ad1f0a25538bb64bad7fb9"} Oct 31 00:48:25.973622 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.973582 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" event={"ID":"1135c40a-4e9e-45e5-9aef-5e4373437998","Type":"ContainerStarted","Data":"32ec9b59dfcc92394ff7f93ac92dcb29ae5886bcbc807ca731540cf477170077"} Oct 31 00:48:25.992186 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:25.992139 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" podStartSLOduration=1.992124756 podStartE2EDuration="1.992124756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:25.990491741 +0000 UTC m=+2990.051266436" watchObservedRunningTime="2026-10-31 00:48:25.992124756 +0000 UTC m=+2990.052899451" Oct 31 00:48:26.019342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.019310 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls"] Oct 31 00:48:26.019484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.019462 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" podUID="d397f027-2786-4b6c-9876-83ef098f31f6" containerName="thanos-querier" containerID="cri-o://a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba" gracePeriod=30 Oct 31 00:48:26.277518 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.277492 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:26.415801 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.415753 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8\" (UniqueName: \"kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8\") pod \"d397f027-2786-4b6c-9876-83ef098f31f6\" (UID: \"d397f027-2786-4b6c-9876-83ef098f31f6\") " Oct 31 00:48:26.417951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.417929 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8" pod "d397f027-2786-4b6c-9876-83ef098f31f6" (UID: "d397f027-2786-4b6c-9876-83ef098f31f6"). InnerVolumeSpecName "kube-api-access-w56x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:26.516983 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.516929 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56x8\" (UniqueName: \"kubernetes.io/projected/d397f027-2786-4b6c-9876-83ef098f31f6-kube-api-access-w56x8\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.978896 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.978863 5073 generic.go:356] "Generic (PLEG): container finished" podID="d397f027-2786-4b6c-9876-83ef098f31f6" containerID="a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba" exitCode=0 Oct 31 00:48:26.979030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.978923 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" Oct 31 00:48:26.979030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.978954 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" event={"ID":"d397f027-2786-4b6c-9876-83ef098f31f6","Type":"ContainerDied","Data":"a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba"} Oct 31 00:48:26.979030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.979006 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls" event={"ID":"d397f027-2786-4b6c-9876-83ef098f31f6","Type":"ContainerDied","Data":"3f674ca7d37f2ba52a9b9bcd049b28bc5804589888725f50a226259e90f22b6b"} Oct 31 00:48:26.979030 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.979024 5073 scope.go:122] "RemoveContainer" containerID="a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba" Oct 31 00:48:26.988104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.988085 5073 scope.go:122] "RemoveContainer" containerID="a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba" Oct 31 00:48:26.988409 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:48:26.988382 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba\": container with ID starting with a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba not found: ID does not exist" containerID="a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba" Oct 31 00:48:26.988513 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:26.988421 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba"} err="failed to get container status \"a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba\": rpc error: code = NotFound desc = could not find container \"a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba\": container with ID starting with a2e43b1c5bcc9f20638c133c59ec258ed8310e37840c7d76fabb3a2f18ebcdba not found: ID does not exist" Oct 31 00:48:27.002835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.002773 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls"] Oct 31 00:48:27.006911 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.006889 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-n22ls"] Oct 31 00:48:27.727580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.727550 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:27.734813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.734774 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:27.828816 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.828769 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:27.828954 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.828892 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:27.832319 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.832302 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:27.922587 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.922561 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:27.938285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:27.938255 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:28.058453 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:28.058419 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d397f027-2786-4b6c-9876-83ef098f31f6" path="/var/lib/kubelet/pods/d397f027-2786-4b6c-9876-83ef098f31f6/volumes" Oct 31 00:48:28.070647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:28.070626 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:28.359880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:28.359804 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:48:28.363085 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:28.363059 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ffbdf0_0f65_44ab_8dfb_e7f5f045815f.slice/crio-b087ab6d659c1e796911e7a8e7ecfe6c25f7f97feb1822e064b7c7b55fb73339 WatchSource:0}: Error finding container b087ab6d659c1e796911e7a8e7ecfe6c25f7f97feb1822e064b7c7b55fb73339: Status 404 returned error can't find the container with id b087ab6d659c1e796911e7a8e7ecfe6c25f7f97feb1822e064b7c7b55fb73339 Oct 31 00:48:28.990799 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:28.990751 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerStarted","Data":"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515"} Oct 31 00:48:28.991138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:28.990840 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerStarted","Data":"b087ab6d659c1e796911e7a8e7ecfe6c25f7f97feb1822e064b7c7b55fb73339"} Oct 31 00:48:30.948802 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:30.948758 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:33.014281 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:33.014191 5073 generic.go:356] "Generic (PLEG): container finished" podID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerID="5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515" exitCode=0 Oct 31 00:48:33.014664 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:33.014272 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerDied","Data":"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515"} Oct 31 00:48:34.022374 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:34.022338 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerStarted","Data":"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e"} Oct 31 00:48:34.022771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:34.022432 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerStarted","Data":"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207"} Oct 31 00:48:34.022771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:34.022457 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerStarted","Data":"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661"} Oct 31 00:48:34.057797 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:34.057738 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=5.872883334 podStartE2EDuration="9.057724453s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.184841119s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:34.055698854 +0000 UTC m=+2998.116473552" watchObservedRunningTime="2026-10-31 00:48:34.057724453 +0000 UTC m=+2998.118499149" Oct 31 00:48:36.381538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:36.381442 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:48:36.410756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:36.388776 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:48:36.410756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:36.391058 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:48:36.410756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:36.398273 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w75k7_3db5e05a-532e-4937-a241-3a995c5c43ae/ovn-acl-logging/0.log" Oct 31 00:48:38.071237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:38.071208 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:40.949280 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:40.949241 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:43.071596 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:43.071563 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:43.077472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:43.077440 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:43.087338 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:48:43.087309 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:48:44.070354 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:44.070328 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:48:46.943906 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:46.943825 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq"] Oct 31 00:48:46.944324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:46.944305 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d397f027-2786-4b6c-9876-83ef098f31f6" containerName="thanos-querier" Oct 31 00:48:46.944386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:46.944329 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d397f027-2786-4b6c-9876-83ef098f31f6" containerName="thanos-querier" Oct 31 00:48:46.944444 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:46.944434 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d397f027-2786-4b6c-9876-83ef098f31f6" containerName="thanos-querier" Oct 31 00:48:46.949151 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:46.949130 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:46.957471 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:46.957187 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq"] Oct 31 00:48:47.090658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.090625 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.090856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.090671 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtkj\" (UniqueName: \"kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.090856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.090751 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.191438 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.191401 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.191621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.191447 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtkj\" (UniqueName: \"kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.191621 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.191562 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.192219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.192189 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.194179 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.194133 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.203930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.203905 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtkj\" (UniqueName: \"kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj\") pod \"data-science-collector-targetallocator-6d8677cbd7-7x2mq\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.266793 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.266761 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:47.539901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.539868 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq"] Oct 31 00:48:47.543072 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:47.543050 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1451459_990d_4f58_9682_2a0a30f28934.slice/crio-1a17a9236a88a17f5e218e884588ff256a084ad743f4c92755dba11b8d5a86da WatchSource:0}: Error finding container 1a17a9236a88a17f5e218e884588ff256a084ad743f4c92755dba11b8d5a86da: Status 404 returned error can't find the container with id 1a17a9236a88a17f5e218e884588ff256a084ad743f4c92755dba11b8d5a86da Oct 31 00:48:47.545180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.545151 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:47.545252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:47.545212 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:48.083849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.083818 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" event={"ID":"d1451459-990d-4f58-9682-2a0a30f28934","Type":"ContainerStarted","Data":"73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c"} Oct 31 00:48:48.083849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.083856 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" event={"ID":"d1451459-990d-4f58-9682-2a0a30f28934","Type":"ContainerStarted","Data":"1a17a9236a88a17f5e218e884588ff256a084ad743f4c92755dba11b8d5a86da"} Oct 31 00:48:48.084362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.083958 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:48.085659 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.085638 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:48:48.105400 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.105345 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" podStartSLOduration=2.105333642 podStartE2EDuration="2.105333642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:48.104470595 +0000 UTC m=+3012.165245290" watchObservedRunningTime="2026-10-31 00:48:48.105333642 +0000 UTC m=+3012.166108370" Oct 31 00:48:48.158535 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.158499 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb"] Oct 31 00:48:48.158705 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.158673 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" podUID="fd07221d-a0fa-48d6-b698-174f08a3bb5c" containerName="ta-container" containerID="cri-o://4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274" gracePeriod=30 Oct 31 00:48:48.431309 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.431286 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:48.608087 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.608056 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal\" (UniqueName: \"kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal\") pod \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " Oct 31 00:48:48.608243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.608166 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert\") pod \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " Oct 31 00:48:48.608243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.608206 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq\" (UniqueName: \"kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq\") pod \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\" (UID: \"fd07221d-a0fa-48d6-b698-174f08a3bb5c\") " Oct 31 00:48:48.608401 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.608379 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal" pod "fd07221d-a0fa-48d6-b698-174f08a3bb5c" (UID: "fd07221d-a0fa-48d6-b698-174f08a3bb5c"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:48.610379 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.610356 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert" pod "fd07221d-a0fa-48d6-b698-174f08a3bb5c" (UID: "fd07221d-a0fa-48d6-b698-174f08a3bb5c"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:48.610497 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.610479 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq" pod "fd07221d-a0fa-48d6-b698-174f08a3bb5c" (UID: "fd07221d-a0fa-48d6-b698-174f08a3bb5c"). InnerVolumeSpecName "kube-api-access-tfvqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:48.709637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.709583 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/fd07221d-a0fa-48d6-b698-174f08a3bb5c-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:48.709637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.709606 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvqq\" (UniqueName: \"kubernetes.io/projected/fd07221d-a0fa-48d6-b698-174f08a3bb5c-kube-api-access-tfvqq\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:48.709637 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:48.709616 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/fd07221d-a0fa-48d6-b698-174f08a3bb5c-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.090132 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.090100 5073 generic.go:356] "Generic (PLEG): container finished" podID="fd07221d-a0fa-48d6-b698-174f08a3bb5c" containerID="4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274" exitCode=0 Oct 31 00:48:49.090576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.090140 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" Oct 31 00:48:49.090576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.090185 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" event={"ID":"fd07221d-a0fa-48d6-b698-174f08a3bb5c","Type":"ContainerDied","Data":"4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274"} Oct 31 00:48:49.090576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.090227 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb" event={"ID":"fd07221d-a0fa-48d6-b698-174f08a3bb5c","Type":"ContainerDied","Data":"652651fdb871f845af2f14fa3a4de5ed658877286a0e83cce964b6da3db86181"} Oct 31 00:48:49.090576 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.090255 5073 scope.go:122] "RemoveContainer" containerID="4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274" Oct 31 00:48:49.099628 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.099611 5073 scope.go:122] "RemoveContainer" containerID="4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274" Oct 31 00:48:49.099957 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:48:49.099937 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274\": container with ID starting with 4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274 not found: ID does not exist" containerID="4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274" Oct 31 00:48:49.100008 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.099967 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274"} err="failed to get container status \"4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274\": rpc error: code = NotFound desc = could not find container \"4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274\": container with ID starting with 4b2c7863a66f0dd5db20519557aaf8b6a082f9f87d53e42886fc49017fad7274 not found: ID does not exist" Oct 31 00:48:49.111771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.111747 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb"] Oct 31 00:48:49.116101 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:49.116080 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-bf55598c4-4cjlb"] Oct 31 00:48:50.058753 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:50.058717 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd07221d-a0fa-48d6-b698-174f08a3bb5c" path="/var/lib/kubelet/pods/fd07221d-a0fa-48d6-b698-174f08a3bb5c/volumes" Oct 31 00:48:50.948875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:50.948833 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:58.950454 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:58.950418 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7"] Oct 31 00:48:58.950935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:58.950771 5073 cpu_manager.go:414] "containerMap: removing container" podUID="fd07221d-a0fa-48d6-b698-174f08a3bb5c" containerName="ta-container" Oct 31 00:48:58.950935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:58.950798 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd07221d-a0fa-48d6-b698-174f08a3bb5c" containerName="ta-container" Oct 31 00:48:58.950935 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:58.950876 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd07221d-a0fa-48d6-b698-174f08a3bb5c" containerName="ta-container" Oct 31 00:48:58.955376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:58.955356 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:58.965647 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:58.965622 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7"] Oct 31 00:48:59.089566 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.089535 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.089700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.089569 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.089700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.089645 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjkp\" (UniqueName: \"kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.190532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.190502 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.190702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.190537 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.190702 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.190576 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjkp\" (UniqueName: \"kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.191202 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.191176 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.192978 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.192955 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.203358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.203311 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjkp\" (UniqueName: \"kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp\") pod \"data-science-collector-targetallocator-5d56cd6c9f-6qkl7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.269944 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.269922 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:48:59.549558 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.549506 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7"] Oct 31 00:48:59.552309 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:48:59.552277 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31e702ca_5c5e_4516_8286_8e9a51f46be7.slice/crio-037a5d7ce49a0d0b4c847299223631f584adf13c35f3257b22d165aa0ebc6af6 WatchSource:0}: Error finding container 037a5d7ce49a0d0b4c847299223631f584adf13c35f3257b22d165aa0ebc6af6: Status 404 returned error can't find the container with id 037a5d7ce49a0d0b4c847299223631f584adf13c35f3257b22d165aa0ebc6af6 Oct 31 00:48:59.557805 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.557747 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:48:59.557919 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:48:59.557832 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:49:00.139342 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.139311 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" event={"ID":"31e702ca-5c5e-4516-8286-8e9a51f46be7","Type":"ContainerStarted","Data":"0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34"} Oct 31 00:49:00.139756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.139352 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" event={"ID":"31e702ca-5c5e-4516-8286-8e9a51f46be7","Type":"ContainerStarted","Data":"037a5d7ce49a0d0b4c847299223631f584adf13c35f3257b22d165aa0ebc6af6"} Oct 31 00:49:00.139756 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.139489 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:49:00.140834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.140814 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:49:00.159519 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.159469 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" podStartSLOduration=2.159453171 podStartE2EDuration="2.159453171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:00.158713843 +0000 UTC m=+3024.219488542" watchObservedRunningTime="2026-10-31 00:49:00.159453171 +0000 UTC m=+3024.220227866" Oct 31 00:49:00.207633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.207599 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq"] Oct 31 00:49:00.207852 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.207827 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" podUID="d1451459-990d-4f58-9682-2a0a30f28934" containerName="ta-container" containerID="cri-o://73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c" gracePeriod=30 Oct 31 00:49:00.467169 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.467143 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:49:00.603609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.603580 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal\" (UniqueName: \"kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal\") pod \"d1451459-990d-4f58-9682-2a0a30f28934\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " Oct 31 00:49:00.603758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.603623 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert\") pod \"d1451459-990d-4f58-9682-2a0a30f28934\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " Oct 31 00:49:00.603758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.603681 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj\" (UniqueName: \"kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj\") pod \"d1451459-990d-4f58-9682-2a0a30f28934\" (UID: \"d1451459-990d-4f58-9682-2a0a30f28934\") " Oct 31 00:49:00.604009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.603990 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal" pod "d1451459-990d-4f58-9682-2a0a30f28934" (UID: "d1451459-990d-4f58-9682-2a0a30f28934"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:00.605923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.605895 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert" pod "d1451459-990d-4f58-9682-2a0a30f28934" (UID: "d1451459-990d-4f58-9682-2a0a30f28934"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:00.606032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.605937 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj" pod "d1451459-990d-4f58-9682-2a0a30f28934" (UID: "d1451459-990d-4f58-9682-2a0a30f28934"). InnerVolumeSpecName "kube-api-access-dgtkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:00.704516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.704463 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/d1451459-990d-4f58-9682-2a0a30f28934-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.704516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.704487 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/d1451459-990d-4f58-9682-2a0a30f28934-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.704516 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.704498 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtkj\" (UniqueName: \"kubernetes.io/projected/d1451459-990d-4f58-9682-2a0a30f28934-kube-api-access-dgtkj\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.949307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:00.949276 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:01.145089 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.145061 5073 generic.go:356] "Generic (PLEG): container finished" podID="d1451459-990d-4f58-9682-2a0a30f28934" containerID="73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c" exitCode=0 Oct 31 00:49:01.145506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.145101 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" Oct 31 00:49:01.145506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.145147 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" event={"ID":"d1451459-990d-4f58-9682-2a0a30f28934","Type":"ContainerDied","Data":"73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c"} Oct 31 00:49:01.145506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.145195 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq" event={"ID":"d1451459-990d-4f58-9682-2a0a30f28934","Type":"ContainerDied","Data":"1a17a9236a88a17f5e218e884588ff256a084ad743f4c92755dba11b8d5a86da"} Oct 31 00:49:01.145506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.145221 5073 scope.go:122] "RemoveContainer" containerID="73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c" Oct 31 00:49:01.154946 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.154928 5073 scope.go:122] "RemoveContainer" containerID="73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c" Oct 31 00:49:01.155179 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:01.155161 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c\": container with ID starting with 73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c not found: ID does not exist" containerID="73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c" Oct 31 00:49:01.155248 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.155187 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c"} err="failed to get container status \"73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c\": rpc error: code = NotFound desc = could not find container \"73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c\": container with ID starting with 73acbda539e73bfc3249ec62cccdeeb4895d6a2e3f3263d18071a05b11b0ed6c not found: ID does not exist" Oct 31 00:49:01.167199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.167175 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq"] Oct 31 00:49:01.171538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:01.171516 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6d8677cbd7-7x2mq"] Oct 31 00:49:02.058701 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:02.058668 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1451459-990d-4f58-9682-2a0a30f28934" path="/var/lib/kubelet/pods/d1451459-990d-4f58-9682-2a0a30f28934/volumes" Oct 31 00:49:04.962192 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:04.962159 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944"] Oct 31 00:49:04.962994 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:04.962945 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" podUID="1135c40a-4e9e-45e5-9aef-5e4373437998" containerName="thanos-querier" containerID="cri-o://0e62d7fd77b70d0b595d9238934610bd2b4386a898ad1f0a25538bb64bad7fb9" gracePeriod=30 Oct 31 00:49:05.063249 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.063213 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:49:05.063763 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.063732 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="prometheus" containerID="cri-o://200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" gracePeriod=600 Oct 31 00:49:05.063947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.063903 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="thanos-sidecar" containerID="cri-o://22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" gracePeriod=600 Oct 31 00:49:05.063947 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.063892 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="config-reloader" containerID="cri-o://b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" gracePeriod=600 Oct 31 00:49:05.167333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.167297 5073 generic.go:356] "Generic (PLEG): container finished" podID="1135c40a-4e9e-45e5-9aef-5e4373437998" containerID="0e62d7fd77b70d0b595d9238934610bd2b4386a898ad1f0a25538bb64bad7fb9" exitCode=0 Oct 31 00:49:05.167470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.167363 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" event={"ID":"1135c40a-4e9e-45e5-9aef-5e4373437998","Type":"ContainerDied","Data":"0e62d7fd77b70d0b595d9238934610bd2b4386a898ad1f0a25538bb64bad7fb9"} Oct 31 00:49:05.276041 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.275995 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7"] Oct 31 00:49:05.276521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.276236 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" podUID="31e702ca-5c5e-4516-8286-8e9a51f46be7" containerName="ta-container" containerID="cri-o://0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34" gracePeriod=30 Oct 31 00:49:05.280552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.280363 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:49:05.344261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.344101 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr\" (UniqueName: \"kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr\") pod \"1135c40a-4e9e-45e5-9aef-5e4373437998\" (UID: \"1135c40a-4e9e-45e5-9aef-5e4373437998\") " Oct 31 00:49:05.347277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.347241 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr" pod "1135c40a-4e9e-45e5-9aef-5e4373437998" (UID: "1135c40a-4e9e-45e5-9aef-5e4373437998"). InnerVolumeSpecName "kube-api-access-ldzwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.415534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.415504 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445021 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445080 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445110 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445143 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445200 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445237 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445320 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445349 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445399 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445451 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445482 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445512 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445609 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\" (UID: \"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f\") " Oct 31 00:49:05.453823 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.445947 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldzwr\" (UniqueName: \"kubernetes.io/projected/1135c40a-4e9e-45e5-9aef-5e4373437998-kube-api-access-ldzwr\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.460658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.455446 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.460658 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.457507 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.466834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.466761 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.467671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.467646 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.469409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.468675 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.473820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.472059 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.473820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.472111 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.473820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.472541 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.473820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.472815 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "kube-api-access-fjknm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.473820 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.473136 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.474466 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.474448 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:49:05.475013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.474976 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.487422 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.487380 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config" pod "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" (UID: "08ffbdf0-0f65-44ab-8dfb-e7f5f045815f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.546856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546760 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.546856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546812 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.546856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546828 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.546856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546846 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546861 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546875 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjknm\" (UniqueName: \"kubernetes.io/projected/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-kube-api-access-fjknm\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546916 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546935 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546953 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546969 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.546982 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.547013 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.547201 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.547028 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.561879 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.561859 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:49:05.608760 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.608736 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:49:05.647584 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.647549 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp\" (UniqueName: \"kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp\") pod \"31e702ca-5c5e-4516-8286-8e9a51f46be7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " Oct 31 00:49:05.647735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.647625 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal\") pod \"31e702ca-5c5e-4516-8286-8e9a51f46be7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " Oct 31 00:49:05.647735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.647726 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert\") pod \"31e702ca-5c5e-4516-8286-8e9a51f46be7\" (UID: \"31e702ca-5c5e-4516-8286-8e9a51f46be7\") " Oct 31 00:49:05.648038 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.648015 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" Oct 31 00:49:05.648196 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.648115 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal" pod "31e702ca-5c5e-4516-8286-8e9a51f46be7" (UID: "31e702ca-5c5e-4516-8286-8e9a51f46be7"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.650054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.650028 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp" pod "31e702ca-5c5e-4516-8286-8e9a51f46be7" (UID: "31e702ca-5c5e-4516-8286-8e9a51f46be7"). InnerVolumeSpecName "kube-api-access-rwjkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.650707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.650681 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert" pod "31e702ca-5c5e-4516-8286-8e9a51f46be7" (UID: "31e702ca-5c5e-4516-8286-8e9a51f46be7"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.748727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.748695 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjkp\" (UniqueName: \"kubernetes.io/projected/31e702ca-5c5e-4516-8286-8e9a51f46be7-kube-api-access-rwjkp\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.748727 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.748724 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e702ca-5c5e-4516-8286-8e9a51f46be7-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.748949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:05.748739 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e702ca-5c5e-4516-8286-8e9a51f46be7-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.172925 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.172895 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" Oct 31 00:49:06.173338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.172900 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944" event={"ID":"1135c40a-4e9e-45e5-9aef-5e4373437998","Type":"ContainerDied","Data":"32ec9b59dfcc92394ff7f93ac92dcb29ae5886bcbc807ca731540cf477170077"} Oct 31 00:49:06.173338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.173026 5073 scope.go:122] "RemoveContainer" containerID="0e62d7fd77b70d0b595d9238934610bd2b4386a898ad1f0a25538bb64bad7fb9" Oct 31 00:49:06.175529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175503 5073 generic.go:356] "Generic (PLEG): container finished" podID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerID="22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" exitCode=0 Oct 31 00:49:06.175529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175528 5073 generic.go:356] "Generic (PLEG): container finished" podID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerID="b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" exitCode=0 Oct 31 00:49:06.175673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175541 5073 generic.go:356] "Generic (PLEG): container finished" podID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerID="200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" exitCode=0 Oct 31 00:49:06.175673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175554 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:06.175673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175577 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerDied","Data":"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e"} Oct 31 00:49:06.175673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175623 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerDied","Data":"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207"} Oct 31 00:49:06.175673 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175654 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerDied","Data":"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661"} Oct 31 00:49:06.175949 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.175686 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"08ffbdf0-0f65-44ab-8dfb-e7f5f045815f","Type":"ContainerDied","Data":"b087ab6d659c1e796911e7a8e7ecfe6c25f7f97feb1822e064b7c7b55fb73339"} Oct 31 00:49:06.177094 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.177069 5073 generic.go:356] "Generic (PLEG): container finished" podID="31e702ca-5c5e-4516-8286-8e9a51f46be7" containerID="0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34" exitCode=0 Oct 31 00:49:06.177224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.177105 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" Oct 31 00:49:06.177224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.177121 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" event={"ID":"31e702ca-5c5e-4516-8286-8e9a51f46be7","Type":"ContainerDied","Data":"0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34"} Oct 31 00:49:06.177224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.177148 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7" event={"ID":"31e702ca-5c5e-4516-8286-8e9a51f46be7","Type":"ContainerDied","Data":"037a5d7ce49a0d0b4c847299223631f584adf13c35f3257b22d165aa0ebc6af6"} Oct 31 00:49:06.182295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.182275 5073 scope.go:122] "RemoveContainer" containerID="22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" Oct 31 00:49:06.190257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.190221 5073 scope.go:122] "RemoveContainer" containerID="b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" Oct 31 00:49:06.196525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.196502 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944"] Oct 31 00:49:06.198817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.198795 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-l6944"] Oct 31 00:49:06.199277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.199264 5073 scope.go:122] "RemoveContainer" containerID="200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" Oct 31 00:49:06.207021 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.207003 5073 scope.go:122] "RemoveContainer" containerID="5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515" Oct 31 00:49:06.212525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.212506 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7"] Oct 31 00:49:06.215048 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.215014 5073 scope.go:122] "RemoveContainer" containerID="22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" Oct 31 00:49:06.215390 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:06.215363 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": container with ID starting with 22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e not found: ID does not exist" containerID="22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" Oct 31 00:49:06.215494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.215397 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e"} err="failed to get container status \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": rpc error: code = NotFound desc = could not find container \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": container with ID starting with 22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e not found: ID does not exist" Oct 31 00:49:06.215494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.215419 5073 scope.go:122] "RemoveContainer" containerID="b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" Oct 31 00:49:06.215775 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:06.215756 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": container with ID starting with b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207 not found: ID does not exist" containerID="b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" Oct 31 00:49:06.215775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.215795 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207"} err="failed to get container status \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": rpc error: code = NotFound desc = could not find container \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": container with ID starting with b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207 not found: ID does not exist" Oct 31 00:49:06.216058 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.215812 5073 scope.go:122] "RemoveContainer" containerID="200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" Oct 31 00:49:06.216108 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:06.216073 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": container with ID starting with 200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661 not found: ID does not exist" containerID="200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" Oct 31 00:49:06.216108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216091 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661"} err="failed to get container status \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": rpc error: code = NotFound desc = could not find container \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": container with ID starting with 200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661 not found: ID does not exist" Oct 31 00:49:06.216108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216102 5073 scope.go:122] "RemoveContainer" containerID="5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515" Oct 31 00:49:06.216336 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:06.216319 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": container with ID starting with 5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515 not found: ID does not exist" containerID="5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515" Oct 31 00:49:06.216386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216341 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515"} err="failed to get container status \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": rpc error: code = NotFound desc = could not find container \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": container with ID starting with 5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515 not found: ID does not exist" Oct 31 00:49:06.216386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216359 5073 scope.go:122] "RemoveContainer" containerID="22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" Oct 31 00:49:06.216610 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216591 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e"} err="failed to get container status \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": rpc error: code = NotFound desc = could not find container \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": container with ID starting with 22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e not found: ID does not exist" Oct 31 00:49:06.216656 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216621 5073 scope.go:122] "RemoveContainer" containerID="b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" Oct 31 00:49:06.216913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216893 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207"} err="failed to get container status \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": rpc error: code = NotFound desc = could not find container \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": container with ID starting with b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207 not found: ID does not exist" Oct 31 00:49:06.216977 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.216913 5073 scope.go:122] "RemoveContainer" containerID="200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" Oct 31 00:49:06.217022 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217005 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-5d56cd6c9f-6qkl7"] Oct 31 00:49:06.217153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217133 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661"} err="failed to get container status \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": rpc error: code = NotFound desc = could not find container \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": container with ID starting with 200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661 not found: ID does not exist" Oct 31 00:49:06.217227 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217155 5073 scope.go:122] "RemoveContainer" containerID="5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515" Oct 31 00:49:06.217367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217350 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515"} err="failed to get container status \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": rpc error: code = NotFound desc = could not find container \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": container with ID starting with 5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515 not found: ID does not exist" Oct 31 00:49:06.217419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217368 5073 scope.go:122] "RemoveContainer" containerID="22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e" Oct 31 00:49:06.217562 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217545 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e"} err="failed to get container status \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": rpc error: code = NotFound desc = could not find container \"22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e\": container with ID starting with 22cfb6306e29456036153bb48f9f4c4803b95e490a30ddc8b7f8b372781f797e not found: ID does not exist" Oct 31 00:49:06.217604 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217562 5073 scope.go:122] "RemoveContainer" containerID="b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207" Oct 31 00:49:06.217757 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217737 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207"} err="failed to get container status \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": rpc error: code = NotFound desc = could not find container \"b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207\": container with ID starting with b408b893c29eac5218b4c1bed69a93ffdef05fb083efdff1c7ba587ed6898207 not found: ID does not exist" Oct 31 00:49:06.217850 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217759 5073 scope.go:122] "RemoveContainer" containerID="200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661" Oct 31 00:49:06.217984 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217966 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661"} err="failed to get container status \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": rpc error: code = NotFound desc = could not find container \"200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661\": container with ID starting with 200ce19355615da5cc925caed72f1ec523e00e206d015d0e7d07b13914afb661 not found: ID does not exist" Oct 31 00:49:06.218054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.217986 5073 scope.go:122] "RemoveContainer" containerID="5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515" Oct 31 00:49:06.218205 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.218187 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515"} err="failed to get container status \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": rpc error: code = NotFound desc = could not find container \"5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515\": container with ID starting with 5066d1956abe82de38e5512432aac6e70aae86420b4f257d520184b234fae515 not found: ID does not exist" Oct 31 00:49:06.218265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.218206 5073 scope.go:122] "RemoveContainer" containerID="0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34" Oct 31 00:49:06.225600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.225583 5073 scope.go:122] "RemoveContainer" containerID="0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34" Oct 31 00:49:06.225965 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:06.225941 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34\": container with ID starting with 0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34 not found: ID does not exist" containerID="0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34" Oct 31 00:49:06.226036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.225965 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34"} err="failed to get container status \"0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34\": rpc error: code = NotFound desc = could not find container \"0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34\": container with ID starting with 0671e7bd38f282c50a8ef8fd3552e6898b8ba9bd711b55fcf384c67d27f9fd34 not found: ID does not exist" Oct 31 00:49:06.230425 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.230375 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:49:06.235120 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:06.235097 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:49:08.059481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:08.059447 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" path="/var/lib/kubelet/pods/08ffbdf0-0f65-44ab-8dfb-e7f5f045815f/volumes" Oct 31 00:49:08.059979 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:08.059960 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1135c40a-4e9e-45e5-9aef-5e4373437998" path="/var/lib/kubelet/pods/1135c40a-4e9e-45e5-9aef-5e4373437998/volumes" Oct 31 00:49:08.060343 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:08.060331 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e702ca-5c5e-4516-8286-8e9a51f46be7" path="/var/lib/kubelet/pods/31e702ca-5c5e-4516-8286-8e9a51f46be7/volumes" Oct 31 00:49:10.949072 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:10.949037 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:13.085425 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:13.085392 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:49:17.391505 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391469 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g"] Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391852 5073 cpu_manager.go:414] "containerMap: removing container" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="init-config-reloader" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391866 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="init-config-reloader" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391875 5073 cpu_manager.go:414] "containerMap: removing container" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="thanos-sidecar" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391880 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="thanos-sidecar" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391893 5073 cpu_manager.go:414] "containerMap: removing container" podUID="1135c40a-4e9e-45e5-9aef-5e4373437998" containerName="thanos-querier" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391898 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1135c40a-4e9e-45e5-9aef-5e4373437998" containerName="thanos-querier" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391908 5073 cpu_manager.go:414] "containerMap: removing container" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="prometheus" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391914 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="prometheus" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391932 5073 cpu_manager.go:414] "containerMap: removing container" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="config-reloader" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391939 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="config-reloader" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391950 5073 cpu_manager.go:414] "containerMap: removing container" podUID="31e702ca-5c5e-4516-8286-8e9a51f46be7" containerName="ta-container" Oct 31 00:49:17.391953 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391957 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e702ca-5c5e-4516-8286-8e9a51f46be7" containerName="ta-container" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391985 5073 cpu_manager.go:414] "containerMap: removing container" podUID="d1451459-990d-4f58-9682-2a0a30f28934" containerName="ta-container" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.391992 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1451459-990d-4f58-9682-2a0a30f28934" containerName="ta-container" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.392068 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="config-reloader" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.392079 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e702ca-5c5e-4516-8286-8e9a51f46be7" containerName="ta-container" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.392091 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1451459-990d-4f58-9682-2a0a30f28934" containerName="ta-container" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.392101 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="thanos-sidecar" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.392111 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="1135c40a-4e9e-45e5-9aef-5e4373437998" containerName="thanos-querier" Oct 31 00:49:17.392359 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.392118 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ffbdf0-0f65-44ab-8dfb-e7f5f045815f" containerName="prometheus" Oct 31 00:49:17.397069 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.397047 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:17.401635 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.401609 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:49:17.403217 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.403192 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g"] Oct 31 00:49:17.437243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.437217 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kz2\" (UniqueName: \"kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g\" (UID: \"c5fb5299-bdcf-435f-98c7-2e876bc70459\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:17.537930 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.537893 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kz2\" (UniqueName: \"kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g\" (UID: \"c5fb5299-bdcf-435f-98c7-2e876bc70459\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:17.551073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.551046 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kz2\" (UniqueName: \"kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g\" (UID: \"c5fb5299-bdcf-435f-98c7-2e876bc70459\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:17.710593 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.710512 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:17.787070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.787036 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj"] Oct 31 00:49:17.792580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.792541 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.805150 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.804829 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj"] Oct 31 00:49:17.841124 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.841092 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrtg\" (UniqueName: \"kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.841276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.841134 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.841276 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.841162 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.898172 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.898135 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:49:17.898356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.898172 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-h8p22\"" Oct 31 00:49:17.898356 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.898241 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:49:17.942368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.942344 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrtg\" (UniqueName: \"kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.942461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.942383 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.942461 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.942418 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.943144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.943117 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.944981 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.944964 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.955070 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.955052 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrtg\" (UniqueName: \"kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg\") pod \"data-science-collector-targetallocator-54dffb86f7-8xswj\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:17.997242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:17.997221 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g"] Oct 31 00:49:18.000330 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:49:18.000310 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5fb5299_bdcf_435f_98c7_2e876bc70459.slice/crio-cd33e6754675db521964422301b0a65d14a55e3727ecfeb980e1084cb4c56028 WatchSource:0}: Error finding container cd33e6754675db521964422301b0a65d14a55e3727ecfeb980e1084cb4c56028: Status 404 returned error can't find the container with id cd33e6754675db521964422301b0a65d14a55e3727ecfeb980e1084cb4c56028 Oct 31 00:49:18.113875 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.113843 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:18.236826 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.236763 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" event={"ID":"c5fb5299-bdcf-435f-98c7-2e876bc70459","Type":"ContainerStarted","Data":"e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1"} Oct 31 00:49:18.236998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.236838 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" event={"ID":"c5fb5299-bdcf-435f-98c7-2e876bc70459","Type":"ContainerStarted","Data":"cd33e6754675db521964422301b0a65d14a55e3727ecfeb980e1084cb4c56028"} Oct 31 00:49:18.254219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.254123 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" podStartSLOduration=1.254108395 podStartE2EDuration="1.254108395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:18.2536729 +0000 UTC m=+3042.314447607" watchObservedRunningTime="2026-10-31 00:49:18.254108395 +0000 UTC m=+3042.314883092" Oct 31 00:49:18.388244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.388206 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj"] Oct 31 00:49:18.390900 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:49:18.390872 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76b49ec8_0299_4e26_823b_d09100809358.slice/crio-028341864dc4d6f65f1bd70daf1a5eeb4bbc78f7d460bb62f6029aae4c68a76d WatchSource:0}: Error finding container 028341864dc4d6f65f1bd70daf1a5eeb4bbc78f7d460bb62f6029aae4c68a76d: Status 404 returned error can't find the container with id 028341864dc4d6f65f1bd70daf1a5eeb4bbc78f7d460bb62f6029aae4c68a76d Oct 31 00:49:18.393163 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.393133 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:49:18.393457 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:18.393188 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:49:19.243417 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:19.243378 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" event={"ID":"76b49ec8-0299-4e26-823b-d09100809358","Type":"ContainerStarted","Data":"b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a"} Oct 31 00:49:19.243620 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:19.243605 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:19.243695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:19.243632 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" event={"ID":"76b49ec8-0299-4e26-823b-d09100809358","Type":"ContainerStarted","Data":"028341864dc4d6f65f1bd70daf1a5eeb4bbc78f7d460bb62f6029aae4c68a76d"} Oct 31 00:49:19.245233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:19.245210 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:19.262695 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:19.262645 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" podStartSLOduration=2.262630834 podStartE2EDuration="2.262630834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:19.262035349 +0000 UTC m=+3043.322810047" watchObservedRunningTime="2026-10-31 00:49:19.262630834 +0000 UTC m=+3043.323405529" Oct 31 00:49:20.949527 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:20.949483 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:23.170121 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.170088 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9"] Oct 31 00:49:23.173105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.173089 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:49:23.185449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.185424 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9"] Oct 31 00:49:23.285289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.285264 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2b92\" (UniqueName: \"kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9\" (UID: \"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:49:23.386333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.386297 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2b92\" (UniqueName: \"kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9\" (UID: \"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:49:23.403057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.403028 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2b92\" (UniqueName: \"kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9\" (UID: \"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:49:23.485303 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.485284 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:49:23.761528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:23.761431 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9"] Oct 31 00:49:23.764644 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:49:23.764616 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea97dcd_5558_46ff_8ffe_1a86d5cd8345.slice/crio-33b659a88c21bb13d9edc022ca58a497d902cb15c4f676aa21142da9185959a3 WatchSource:0}: Error finding container 33b659a88c21bb13d9edc022ca58a497d902cb15c4f676aa21142da9185959a3: Status 404 returned error can't find the container with id 33b659a88c21bb13d9edc022ca58a497d902cb15c4f676aa21142da9185959a3 Oct 31 00:49:24.123324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.123248 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:49:24.127082 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.127063 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.131042 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131017 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:49:24.131269 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131060 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:49:24.131269 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131131 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:49:24.131402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131271 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:49:24.131402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131296 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-p8r6g\"" Oct 31 00:49:24.131402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131307 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:49:24.131402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131309 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:49:24.131402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.131365 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:49:24.132199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.132178 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:49:24.132310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.132204 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:49:24.145346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.145314 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:49:24.194368 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194338 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.194734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194384 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxj65\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.194734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194519 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.194734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194559 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.194734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194600 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.194734 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194627 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.195017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194811 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.195017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194840 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.195017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194865 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.195017 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.194909 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.195168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.195007 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.195168 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.195047 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.267257 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.267228 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" event={"ID":"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345","Type":"ContainerStarted","Data":"a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f"} Oct 31 00:49:24.267378 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.267273 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" event={"ID":"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345","Type":"ContainerStarted","Data":"33b659a88c21bb13d9edc022ca58a497d902cb15c4f676aa21142da9185959a3"} Oct 31 00:49:24.286767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.286716 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" podStartSLOduration=1.286699691 podStartE2EDuration="1.286699691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:24.286562518 +0000 UTC m=+3048.347337213" watchObservedRunningTime="2026-10-31 00:49:24.286699691 +0000 UTC m=+3048.347474389" Oct 31 00:49:24.295958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.295933 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.295972 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.295991 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296027 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296061 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296081 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296099 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296460 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296129 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296460 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296170 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296460 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296242 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296460 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296271 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxj65\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296460 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296315 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.296747 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.296714 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.297062 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.297038 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.297813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.297753 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.300296 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.299926 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.301045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.300831 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.301045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.301003 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.301333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.301309 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.301333 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.301327 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.301536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.301514 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.301729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.301713 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.302589 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.302570 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.311284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.311262 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxj65\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:24.313216 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.313198 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g"] Oct 31 00:49:24.313361 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.313344 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" podUID="c5fb5299-bdcf-435f-98c7-2e876bc70459" containerName="thanos-querier" containerID="cri-o://e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1" gracePeriod=30 Oct 31 00:49:24.562803 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.562764 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:24.599318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.599292 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2\" (UniqueName: \"kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2\") pod \"c5fb5299-bdcf-435f-98c7-2e876bc70459\" (UID: \"c5fb5299-bdcf-435f-98c7-2e876bc70459\") " Oct 31 00:49:24.601468 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.601448 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2" pod "c5fb5299-bdcf-435f-98c7-2e876bc70459" (UID: "c5fb5299-bdcf-435f-98c7-2e876bc70459"). InnerVolumeSpecName "kube-api-access-h2kz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:24.700540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:24.700486 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2kz2\" (UniqueName: \"kubernetes.io/projected/c5fb5299-bdcf-435f-98c7-2e876bc70459-kube-api-access-h2kz2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:25.272993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.272962 5073 generic.go:356] "Generic (PLEG): container finished" podID="c5fb5299-bdcf-435f-98c7-2e876bc70459" containerID="e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1" exitCode=0 Oct 31 00:49:25.273428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.273011 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" Oct 31 00:49:25.273428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.273053 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" event={"ID":"c5fb5299-bdcf-435f-98c7-2e876bc70459","Type":"ContainerDied","Data":"e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1"} Oct 31 00:49:25.273428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.273098 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g" event={"ID":"c5fb5299-bdcf-435f-98c7-2e876bc70459","Type":"ContainerDied","Data":"cd33e6754675db521964422301b0a65d14a55e3727ecfeb980e1084cb4c56028"} Oct 31 00:49:25.273428 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.273116 5073 scope.go:122] "RemoveContainer" containerID="e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1" Oct 31 00:49:25.282528 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.282512 5073 scope.go:122] "RemoveContainer" containerID="e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1" Oct 31 00:49:25.282813 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:25.282766 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1\": container with ID starting with e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1 not found: ID does not exist" containerID="e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1" Oct 31 00:49:25.282908 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.282840 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1"} err="failed to get container status \"e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1\": rpc error: code = NotFound desc = could not find container \"e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1\": container with ID starting with e8d675d247fa3430ed9e67916c6dd32e9ac6098e6236884af913346d5c7772f1 not found: ID does not exist" Oct 31 00:49:25.295073 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.295054 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g"] Oct 31 00:49:25.300323 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:25.300300 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-dts2g"] Oct 31 00:49:26.066032 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:26.065994 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fb5299-bdcf-435f-98c7-2e876bc70459" path="/var/lib/kubelet/pods/c5fb5299-bdcf-435f-98c7-2e876bc70459/volumes" Oct 31 00:49:27.424054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.424022 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.430750 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.430726 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.525198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.525170 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.525337 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.525316 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.530831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.530774 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.622095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.622067 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.639327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.639306 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:27.748534 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:27.748512 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:28.041619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:28.041595 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:49:28.044196 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:49:28.044166 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa6285f_8d00_4c2b_9cfc_db454d834972.slice/crio-abd89f577c3b772f03dea1307d58634bd97c58af43eebdfc28f0e82a44f6dac2 WatchSource:0}: Error finding container abd89f577c3b772f03dea1307d58634bd97c58af43eebdfc28f0e82a44f6dac2: Status 404 returned error can't find the container with id abd89f577c3b772f03dea1307d58634bd97c58af43eebdfc28f0e82a44f6dac2 Oct 31 00:49:28.288554 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:28.288523 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerStarted","Data":"abd89f577c3b772f03dea1307d58634bd97c58af43eebdfc28f0e82a44f6dac2"} Oct 31 00:49:29.294676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:29.294642 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerStarted","Data":"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c"} Oct 31 00:49:30.949536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:30.949502 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:32.316802 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:32.316757 5073 generic.go:356] "Generic (PLEG): container finished" podID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerID="cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c" exitCode=0 Oct 31 00:49:32.317237 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:32.316828 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerDied","Data":"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c"} Oct 31 00:49:33.323872 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:33.323836 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerStarted","Data":"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996"} Oct 31 00:49:33.324304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:33.323903 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerStarted","Data":"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6"} Oct 31 00:49:33.324304 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:33.323940 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerStarted","Data":"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961"} Oct 31 00:49:33.358092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:33.358039 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=6.172312471 podStartE2EDuration="9.358024339s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.185711868s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:33.35653186 +0000 UTC m=+3057.417306556" watchObservedRunningTime="2026-10-31 00:49:33.358024339 +0000 UTC m=+3057.418799035" Oct 31 00:49:37.748832 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:37.748804 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:40.948556 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:40.948520 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:42.749635 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:42.749595 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:42.755012 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:42.754990 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:43.085095 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:43.085021 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:49:43.374697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:43.374624 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:49:46.077006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.076969 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29"] Oct 31 00:49:46.077487 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.077468 5073 cpu_manager.go:414] "containerMap: removing container" podUID="c5fb5299-bdcf-435f-98c7-2e876bc70459" containerName="thanos-querier" Oct 31 00:49:46.077536 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.077489 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fb5299-bdcf-435f-98c7-2e876bc70459" containerName="thanos-querier" Oct 31 00:49:46.077580 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.077545 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fb5299-bdcf-435f-98c7-2e876bc70459" containerName="thanos-querier" Oct 31 00:49:46.082219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.082194 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.091914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.091886 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29"] Oct 31 00:49:46.277512 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.277480 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vm4\" (UniqueName: \"kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.277683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.277539 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.277683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.277620 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.378289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.378217 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.378423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.378287 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.378423 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.378347 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vm4\" (UniqueName: \"kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.378937 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.378911 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.381399 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.381371 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.391295 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.391275 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vm4\" (UniqueName: \"kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4\") pod \"data-science-collector-targetallocator-6f6d7c6c96-t7s29\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.397109 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.397088 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:46.674338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.674260 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29"] Oct 31 00:49:46.676895 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:49:46.676870 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod662a69f1_ae92_47c8_b1fa_2de72ba91cef.slice/crio-f28f19b8fba177376aeebd5603f8a1fa1277477f78805083e3ab4f255e294ed3 WatchSource:0}: Error finding container f28f19b8fba177376aeebd5603f8a1fa1277477f78805083e3ab4f255e294ed3: Status 404 returned error can't find the container with id f28f19b8fba177376aeebd5603f8a1fa1277477f78805083e3ab4f255e294ed3 Oct 31 00:49:46.679274 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.679232 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:49:46.679370 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:46.679315 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:49:47.390383 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.390344 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" event={"ID":"662a69f1-ae92-47c8-b1fa-2de72ba91cef","Type":"ContainerStarted","Data":"969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1"} Oct 31 00:49:47.390830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.390389 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" event={"ID":"662a69f1-ae92-47c8-b1fa-2de72ba91cef","Type":"ContainerStarted","Data":"f28f19b8fba177376aeebd5603f8a1fa1277477f78805083e3ab4f255e294ed3"} Oct 31 00:49:47.390830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.390559 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:47.392411 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.392389 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:49:47.430285 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.430233 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" podStartSLOduration=1.430220504 podStartE2EDuration="1.430220504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:47.40961324 +0000 UTC m=+3071.470387937" watchObservedRunningTime="2026-10-31 00:49:47.430220504 +0000 UTC m=+3071.490995198" Oct 31 00:49:47.460130 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.460100 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj"] Oct 31 00:49:47.460299 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.460274 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" podUID="76b49ec8-0299-4e26-823b-d09100809358" containerName="ta-container" containerID="cri-o://b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a" gracePeriod=30 Oct 31 00:49:47.749365 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.749329 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:47.802808 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.802761 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert\") pod \"76b49ec8-0299-4e26-823b-d09100809358\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " Oct 31 00:49:47.803005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.802818 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg\" (UniqueName: \"kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg\") pod \"76b49ec8-0299-4e26-823b-d09100809358\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " Oct 31 00:49:47.803005 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.802909 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal\" (UniqueName: \"kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal\") pod \"76b49ec8-0299-4e26-823b-d09100809358\" (UID: \"76b49ec8-0299-4e26-823b-d09100809358\") " Oct 31 00:49:47.803302 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.803268 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal" pod "76b49ec8-0299-4e26-823b-d09100809358" (UID: "76b49ec8-0299-4e26-823b-d09100809358"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:47.805117 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.805096 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg" pod "76b49ec8-0299-4e26-823b-d09100809358" (UID: "76b49ec8-0299-4e26-823b-d09100809358"). InnerVolumeSpecName "kube-api-access-hnrtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:47.805213 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.805115 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert" pod "76b49ec8-0299-4e26-823b-d09100809358" (UID: "76b49ec8-0299-4e26-823b-d09100809358"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:47.903565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.903529 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/76b49ec8-0299-4e26-823b-d09100809358-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:47.903565 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.903561 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/76b49ec8-0299-4e26-823b-d09100809358-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:47.903772 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:47.903575 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrtg\" (UniqueName: \"kubernetes.io/projected/76b49ec8-0299-4e26-823b-d09100809358-kube-api-access-hnrtg\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:48.396484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.396396 5073 generic.go:356] "Generic (PLEG): container finished" podID="76b49ec8-0299-4e26-823b-d09100809358" containerID="b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a" exitCode=0 Oct 31 00:49:48.396484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.396446 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" Oct 31 00:49:48.397054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.396476 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" event={"ID":"76b49ec8-0299-4e26-823b-d09100809358","Type":"ContainerDied","Data":"b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a"} Oct 31 00:49:48.397054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.396532 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj" event={"ID":"76b49ec8-0299-4e26-823b-d09100809358","Type":"ContainerDied","Data":"028341864dc4d6f65f1bd70daf1a5eeb4bbc78f7d460bb62f6029aae4c68a76d"} Oct 31 00:49:48.397054 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.396570 5073 scope.go:122] "RemoveContainer" containerID="b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a" Oct 31 00:49:48.405326 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.405298 5073 scope.go:122] "RemoveContainer" containerID="b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a" Oct 31 00:49:48.405608 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:49:48.405591 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a\": container with ID starting with b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a not found: ID does not exist" containerID="b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a" Oct 31 00:49:48.405661 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.405619 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a"} err="failed to get container status \"b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a\": rpc error: code = NotFound desc = could not find container \"b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a\": container with ID starting with b76afba75d68fbafc1f6c58a4538a301eb81d6b123395de4a56aea6488b99f7a not found: ID does not exist" Oct 31 00:49:48.415178 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.415155 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj"] Oct 31 00:49:48.419700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:48.419678 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-54dffb86f7-8xswj"] Oct 31 00:49:50.058463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:50.058424 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b49ec8-0299-4e26-823b-d09100809358" path="/var/lib/kubelet/pods/76b49ec8-0299-4e26-823b-d09100809358/volumes" Oct 31 00:49:50.949515 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:49:50.949467 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:00.949375 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:00.949338 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:03.634660 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.634620 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph"] Oct 31 00:50:03.635066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.635011 5073 cpu_manager.go:414] "containerMap: removing container" podUID="76b49ec8-0299-4e26-823b-d09100809358" containerName="ta-container" Oct 31 00:50:03.635066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.635023 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b49ec8-0299-4e26-823b-d09100809358" containerName="ta-container" Oct 31 00:50:03.635140 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.635082 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b49ec8-0299-4e26-823b-d09100809358" containerName="ta-container" Oct 31 00:50:03.639482 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.639459 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.647829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.647762 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph"] Oct 31 00:50:03.723006 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.722976 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.723144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.723040 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggcx\" (UniqueName: \"kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.723144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.723105 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.824289 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.824262 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.824419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.824320 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggcx\" (UniqueName: \"kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.824419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.824339 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.824864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.824840 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.826848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.826823 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.837083 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.837062 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggcx\" (UniqueName: \"kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx\") pod \"data-science-collector-targetallocator-599d687dc9-sghph\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:03.955046 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:03.954974 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:04.234206 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.234184 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph"] Oct 31 00:50:04.236979 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:50:04.236953 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99ee68b3_d021_4dcc_9e97_c7f764ed986d.slice/crio-4231889e783219ccc8bda429d8d6e8dc39e9cfe0f9ea6bc376c649cfbfe821a4 WatchSource:0}: Error finding container 4231889e783219ccc8bda429d8d6e8dc39e9cfe0f9ea6bc376c649cfbfe821a4: Status 404 returned error can't find the container with id 4231889e783219ccc8bda429d8d6e8dc39e9cfe0f9ea6bc376c649cfbfe821a4 Oct 31 00:50:04.239181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.239150 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:50:04.239259 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.239205 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:50:04.470683 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.470647 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" event={"ID":"99ee68b3-d021-4dcc-9e97-c7f764ed986d","Type":"ContainerStarted","Data":"aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e"} Oct 31 00:50:04.470901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.470698 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" event={"ID":"99ee68b3-d021-4dcc-9e97-c7f764ed986d","Type":"ContainerStarted","Data":"4231889e783219ccc8bda429d8d6e8dc39e9cfe0f9ea6bc376c649cfbfe821a4"} Oct 31 00:50:04.470901 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.470839 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:04.472532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.472512 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:04.492026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.491942 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" podStartSLOduration=1.49190439 podStartE2EDuration="1.49190439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:04.491500124 +0000 UTC m=+3088.552274821" watchObservedRunningTime="2026-10-31 00:50:04.49190439 +0000 UTC m=+3088.552679086" Oct 31 00:50:04.541951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.541924 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29"] Oct 31 00:50:04.542104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.542081 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" podUID="662a69f1-ae92-47c8-b1fa-2de72ba91cef" containerName="ta-container" containerID="cri-o://969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1" gracePeriod=30 Oct 31 00:50:04.817105 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.817077 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:50:04.933904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.933877 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal\" (UniqueName: \"kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal\") pod \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " Oct 31 00:50:04.934055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.933931 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert\") pod \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " Oct 31 00:50:04.934055 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.933954 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4\" (UniqueName: \"kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4\") pod \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\" (UID: \"662a69f1-ae92-47c8-b1fa-2de72ba91cef\") " Oct 31 00:50:04.934346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.934308 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal" pod "662a69f1-ae92-47c8-b1fa-2de72ba91cef" (UID: "662a69f1-ae92-47c8-b1fa-2de72ba91cef"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:04.936226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.936204 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4" pod "662a69f1-ae92-47c8-b1fa-2de72ba91cef" (UID: "662a69f1-ae92-47c8-b1fa-2de72ba91cef"). InnerVolumeSpecName "kube-api-access-j7vm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:04.936328 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:04.936242 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert" pod "662a69f1-ae92-47c8-b1fa-2de72ba91cef" (UID: "662a69f1-ae92-47c8-b1fa-2de72ba91cef"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:05.035034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.035013 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/662a69f1-ae92-47c8-b1fa-2de72ba91cef-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:05.035034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.035035 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7vm4\" (UniqueName: \"kubernetes.io/projected/662a69f1-ae92-47c8-b1fa-2de72ba91cef-kube-api-access-j7vm4\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:05.035188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.035045 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/662a69f1-ae92-47c8-b1fa-2de72ba91cef-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:05.476849 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.476816 5073 generic.go:356] "Generic (PLEG): container finished" podID="662a69f1-ae92-47c8-b1fa-2de72ba91cef" containerID="969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1" exitCode=0 Oct 31 00:50:05.477014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.476902 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" Oct 31 00:50:05.477014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.476948 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" event={"ID":"662a69f1-ae92-47c8-b1fa-2de72ba91cef","Type":"ContainerDied","Data":"969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1"} Oct 31 00:50:05.477014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.476985 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29" event={"ID":"662a69f1-ae92-47c8-b1fa-2de72ba91cef","Type":"ContainerDied","Data":"f28f19b8fba177376aeebd5603f8a1fa1277477f78805083e3ab4f255e294ed3"} Oct 31 00:50:05.477014 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.477004 5073 scope.go:122] "RemoveContainer" containerID="969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1" Oct 31 00:50:05.486040 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.486023 5073 scope.go:122] "RemoveContainer" containerID="969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1" Oct 31 00:50:05.486322 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:05.486301 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1\": container with ID starting with 969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1 not found: ID does not exist" containerID="969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1" Oct 31 00:50:05.486406 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.486329 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1"} err="failed to get container status \"969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1\": rpc error: code = NotFound desc = could not find container \"969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1\": container with ID starting with 969a325a23e139d36ef1850d27cb5bd6a5246f1f48849fae23200f66fc154ba1 not found: ID does not exist" Oct 31 00:50:05.500263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.500240 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29"] Oct 31 00:50:05.504657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:05.504637 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6f6d7c6c96-t7s29"] Oct 31 00:50:06.058066 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:06.058038 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662a69f1-ae92-47c8-b1fa-2de72ba91cef" path="/var/lib/kubelet/pods/662a69f1-ae92-47c8-b1fa-2de72ba91cef/volumes" Oct 31 00:50:09.428114 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.428074 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9"] Oct 31 00:50:09.428998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.428952 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" podUID="4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" containerName="thanos-querier" containerID="cri-o://a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f" gracePeriod=30 Oct 31 00:50:09.513600 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.513569 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:09.513924 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.513873 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="prometheus" containerID="cri-o://1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" gracePeriod=600 Oct 31 00:50:09.514108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.514082 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="thanos-sidecar" containerID="cri-o://bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" gracePeriod=600 Oct 31 00:50:09.514196 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.514088 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="config-reloader" containerID="cri-o://48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" gracePeriod=600 Oct 31 00:50:09.725389 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.725356 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph"] Oct 31 00:50:09.725542 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.725522 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" podUID="99ee68b3-d021-4dcc-9e97-c7f764ed986d" containerName="ta-container" containerID="cri-o://aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e" gracePeriod=30 Oct 31 00:50:09.934144 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.933889 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:50:09.973188 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.973107 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92\" (UniqueName: \"kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92\") pod \"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345\" (UID: \"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345\") " Oct 31 00:50:09.975627 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:09.975598 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92" pod "4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" (UID: "4ea97dcd-5558-46ff-8ffe-1a86d5cd8345"). InnerVolumeSpecName "kube-api-access-p2b92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.074856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.074828 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2b92\" (UniqueName: \"kubernetes.io/projected/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345-kube-api-access-p2b92\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.144265 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.144238 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:10.146402 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.146371 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.175907 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.175972 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176003 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176071 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176104 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176143 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176177 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176221 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176279 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176315 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176343 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176383 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.177353 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.176486 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"7fa6285f-8d00-4c2b-9cfc-db454d834972\" (UID: \"7fa6285f-8d00-4c2b-9cfc-db454d834972\") " Oct 31 00:50:10.178571 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.178459 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.179362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.179338 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.180098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.179770 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.180098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.180004 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.183904 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.183877 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.184010 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.183922 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.184071 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.184043 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "kube-api-access-cxj65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.184197 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.184171 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.184521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.184479 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.184636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.184520 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.184958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.184933 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.188045 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.188021 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:50:10.197263 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.197224 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config" pod "7fa6285f-8d00-4c2b-9cfc-db454d834972" (UID: "7fa6285f-8d00-4c2b-9cfc-db454d834972"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.277059 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277022 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert\") pod \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " Oct 31 00:50:10.277226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277092 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal\" (UniqueName: \"kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal\") pod \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " Oct 31 00:50:10.277226 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277118 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx\" (UniqueName: \"kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx\") pod \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\" (UID: \"99ee68b3-d021-4dcc-9e97-c7f764ed986d\") " Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277379 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277401 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277415 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277430 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277466 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277481 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7fa6285f-8d00-4c2b-9cfc-db454d834972-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277496 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7fa6285f-8d00-4c2b-9cfc-db454d834972-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277511 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxj65\" (UniqueName: \"kubernetes.io/projected/7fa6285f-8d00-4c2b-9cfc-db454d834972-kube-api-access-cxj65\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277526 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277540 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277552 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277565 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277578 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7fa6285f-8d00-4c2b-9cfc-db454d834972-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.277619 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.277504 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal" pod "99ee68b3-d021-4dcc-9e97-c7f764ed986d" (UID: "99ee68b3-d021-4dcc-9e97-c7f764ed986d"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.280049 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.280024 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert" pod "99ee68b3-d021-4dcc-9e97-c7f764ed986d" (UID: "99ee68b3-d021-4dcc-9e97-c7f764ed986d"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.280418 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.280393 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx" pod "99ee68b3-d021-4dcc-9e97-c7f764ed986d" (UID: "99ee68b3-d021-4dcc-9e97-c7f764ed986d"). InnerVolumeSpecName "kube-api-access-sggcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.289858 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.289841 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:50:10.378959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.378932 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/99ee68b3-d021-4dcc-9e97-c7f764ed986d-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.378959 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.378959 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/99ee68b3-d021-4dcc-9e97-c7f764ed986d-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.379153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.378973 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggcx\" (UniqueName: \"kubernetes.io/projected/99ee68b3-d021-4dcc-9e97-c7f764ed986d-kube-api-access-sggcx\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.379153 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.378990 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" Oct 31 00:50:10.502100 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.502064 5073 generic.go:356] "Generic (PLEG): container finished" podID="4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" containerID="a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f" exitCode=0 Oct 31 00:50:10.502552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.502103 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" Oct 31 00:50:10.502552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.502139 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" event={"ID":"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345","Type":"ContainerDied","Data":"a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f"} Oct 31 00:50:10.502552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.502187 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9" event={"ID":"4ea97dcd-5558-46ff-8ffe-1a86d5cd8345","Type":"ContainerDied","Data":"33b659a88c21bb13d9edc022ca58a497d902cb15c4f676aa21142da9185959a3"} Oct 31 00:50:10.502552 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.502215 5073 scope.go:122] "RemoveContainer" containerID="a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f" Oct 31 00:50:10.503735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.503713 5073 generic.go:356] "Generic (PLEG): container finished" podID="99ee68b3-d021-4dcc-9e97-c7f764ed986d" containerID="aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e" exitCode=0 Oct 31 00:50:10.503990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.503764 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" Oct 31 00:50:10.503990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.503821 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" event={"ID":"99ee68b3-d021-4dcc-9e97-c7f764ed986d","Type":"ContainerDied","Data":"aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e"} Oct 31 00:50:10.503990 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.503871 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph" event={"ID":"99ee68b3-d021-4dcc-9e97-c7f764ed986d","Type":"ContainerDied","Data":"4231889e783219ccc8bda429d8d6e8dc39e9cfe0f9ea6bc376c649cfbfe821a4"} Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507342 5073 generic.go:356] "Generic (PLEG): container finished" podID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerID="bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" exitCode=0 Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507368 5073 generic.go:356] "Generic (PLEG): container finished" podID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerID="48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" exitCode=0 Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507377 5073 generic.go:356] "Generic (PLEG): container finished" podID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerID="1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" exitCode=0 Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507404 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerDied","Data":"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996"} Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507478 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerDied","Data":"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6"} Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507527 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerDied","Data":"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961"} Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507567 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7fa6285f-8d00-4c2b-9cfc-db454d834972","Type":"ContainerDied","Data":"abd89f577c3b772f03dea1307d58634bd97c58af43eebdfc28f0e82a44f6dac2"} Oct 31 00:50:10.507367 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.507411 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:10.513482 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.513282 5073 scope.go:122] "RemoveContainer" containerID="a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f" Oct 31 00:50:10.513650 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:10.513623 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f\": container with ID starting with a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f not found: ID does not exist" containerID="a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f" Oct 31 00:50:10.513728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.513660 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f"} err="failed to get container status \"a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f\": rpc error: code = NotFound desc = could not find container \"a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f\": container with ID starting with a903fd0c95a67f24c9299de35bcd907d432af2b9eed8484d5498b750dd2ab06f not found: ID does not exist" Oct 31 00:50:10.513728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.513682 5073 scope.go:122] "RemoveContainer" containerID="aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e" Oct 31 00:50:10.522998 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.522939 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9"] Oct 31 00:50:10.526431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.525103 5073 scope.go:122] "RemoveContainer" containerID="aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e" Oct 31 00:50:10.526431 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:10.525463 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e\": container with ID starting with aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e not found: ID does not exist" containerID="aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e" Oct 31 00:50:10.526431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.525493 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e"} err="failed to get container status \"aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e\": rpc error: code = NotFound desc = could not find container \"aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e\": container with ID starting with aae5105e4451537832c61f475f34099df2af06343987e01f791b6765df1fe46e not found: ID does not exist" Oct 31 00:50:10.526431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.525512 5073 scope.go:122] "RemoveContainer" containerID="bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" Oct 31 00:50:10.527994 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.527947 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7qfj9"] Oct 31 00:50:10.535640 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.535620 5073 scope.go:122] "RemoveContainer" containerID="48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" Oct 31 00:50:10.544224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.544203 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph"] Oct 31 00:50:10.545125 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.545113 5073 scope.go:122] "RemoveContainer" containerID="1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" Oct 31 00:50:10.548835 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.548814 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-599d687dc9-sghph"] Oct 31 00:50:10.553696 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.553681 5073 scope.go:122] "RemoveContainer" containerID="cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c" Oct 31 00:50:10.561913 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.561897 5073 scope.go:122] "RemoveContainer" containerID="bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" Oct 31 00:50:10.562217 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:10.562196 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": container with ID starting with bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996 not found: ID does not exist" containerID="bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" Oct 31 00:50:10.562332 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562224 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996"} err="failed to get container status \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": rpc error: code = NotFound desc = could not find container \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": container with ID starting with bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996 not found: ID does not exist" Oct 31 00:50:10.562332 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562240 5073 scope.go:122] "RemoveContainer" containerID="48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" Oct 31 00:50:10.562459 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:10.562439 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": container with ID starting with 48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6 not found: ID does not exist" containerID="48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" Oct 31 00:50:10.562499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562464 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6"} err="failed to get container status \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": rpc error: code = NotFound desc = could not find container \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": container with ID starting with 48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6 not found: ID does not exist" Oct 31 00:50:10.562499 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562479 5073 scope.go:122] "RemoveContainer" containerID="1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" Oct 31 00:50:10.562714 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:10.562696 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": container with ID starting with 1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961 not found: ID does not exist" containerID="1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" Oct 31 00:50:10.562771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562716 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961"} err="failed to get container status \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": rpc error: code = NotFound desc = could not find container \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": container with ID starting with 1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961 not found: ID does not exist" Oct 31 00:50:10.562771 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562727 5073 scope.go:122] "RemoveContainer" containerID="cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c" Oct 31 00:50:10.562921 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562902 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:10.562963 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:10.562939 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": container with ID starting with cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c not found: ID does not exist" containerID="cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c" Oct 31 00:50:10.563009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562961 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c"} err="failed to get container status \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": rpc error: code = NotFound desc = could not find container \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": container with ID starting with cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c not found: ID does not exist" Oct 31 00:50:10.563009 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.562976 5073 scope.go:122] "RemoveContainer" containerID="bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" Oct 31 00:50:10.563244 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.563219 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996"} err="failed to get container status \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": rpc error: code = NotFound desc = could not find container \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": container with ID starting with bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996 not found: ID does not exist" Oct 31 00:50:10.563308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.563247 5073 scope.go:122] "RemoveContainer" containerID="48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" Oct 31 00:50:10.563553 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.563530 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6"} err="failed to get container status \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": rpc error: code = NotFound desc = could not find container \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": container with ID starting with 48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6 not found: ID does not exist" Oct 31 00:50:10.563631 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.563554 5073 scope.go:122] "RemoveContainer" containerID="1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" Oct 31 00:50:10.563827 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.563805 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961"} err="failed to get container status \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": rpc error: code = NotFound desc = could not find container \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": container with ID starting with 1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961 not found: ID does not exist" Oct 31 00:50:10.563912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.563828 5073 scope.go:122] "RemoveContainer" containerID="cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c" Oct 31 00:50:10.564092 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564072 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c"} err="failed to get container status \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": rpc error: code = NotFound desc = could not find container \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": container with ID starting with cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c not found: ID does not exist" Oct 31 00:50:10.564147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564094 5073 scope.go:122] "RemoveContainer" containerID="bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996" Oct 31 00:50:10.564345 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564324 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996"} err="failed to get container status \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": rpc error: code = NotFound desc = could not find container \"bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996\": container with ID starting with bf19ea8289d5a36e1bd6db87da2eefe7bd0fd013ca15f82d2d97f3e9989fe996 not found: ID does not exist" Oct 31 00:50:10.564409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564347 5073 scope.go:122] "RemoveContainer" containerID="48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6" Oct 31 00:50:10.564609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564587 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6"} err="failed to get container status \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": rpc error: code = NotFound desc = could not find container \"48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6\": container with ID starting with 48783b05efcda420670bb957af234c4668134a0f30752b73e271797901ba1bc6 not found: ID does not exist" Oct 31 00:50:10.564609 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564607 5073 scope.go:122] "RemoveContainer" containerID="1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961" Oct 31 00:50:10.564887 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564863 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961"} err="failed to get container status \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": rpc error: code = NotFound desc = could not find container \"1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961\": container with ID starting with 1131ce0102f8ba34c72ea0481c39f7a2ccde93f9089208686208363a9216b961 not found: ID does not exist" Oct 31 00:50:10.564887 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.564884 5073 scope.go:122] "RemoveContainer" containerID="cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c" Oct 31 00:50:10.565121 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.565104 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c"} err="failed to get container status \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": rpc error: code = NotFound desc = could not find container \"cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c\": container with ID starting with cd20c128bee4d0b073642a0498333682e920c13738c35ff8f3bf49715b43110c not found: ID does not exist" Oct 31 00:50:10.570314 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.570291 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:10.949729 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:10.949645 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:12.058800 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:12.058753 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" path="/var/lib/kubelet/pods/4ea97dcd-5558-46ff-8ffe-1a86d5cd8345/volumes" Oct 31 00:50:12.059243 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:12.059226 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" path="/var/lib/kubelet/pods/7fa6285f-8d00-4c2b-9cfc-db454d834972/volumes" Oct 31 00:50:12.059728 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:12.059712 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ee68b3-d021-4dcc-9e97-c7f764ed986d" path="/var/lib/kubelet/pods/99ee68b3-d021-4dcc-9e97-c7f764ed986d/volumes" Oct 31 00:50:13.084624 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:13.084589 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:50:20.274147 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274058 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj"] Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274542 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="init-config-reloader" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274561 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="init-config-reloader" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274575 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="config-reloader" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274583 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="config-reloader" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274607 5073 cpu_manager.go:414] "containerMap: removing container" podUID="4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" containerName="thanos-querier" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274615 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" containerName="thanos-querier" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274628 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="prometheus" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274634 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="prometheus" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274649 5073 cpu_manager.go:414] "containerMap: removing container" podUID="99ee68b3-d021-4dcc-9e97-c7f764ed986d" containerName="ta-container" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274656 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ee68b3-d021-4dcc-9e97-c7f764ed986d" containerName="ta-container" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274678 5073 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="thanos-sidecar" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274685 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="thanos-sidecar" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274699 5073 cpu_manager.go:414] "containerMap: removing container" podUID="662a69f1-ae92-47c8-b1fa-2de72ba91cef" containerName="ta-container" Oct 31 00:50:20.274730 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274707 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662a69f1-ae92-47c8-b1fa-2de72ba91cef" containerName="ta-container" Oct 31 00:50:20.275545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274817 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="thanos-sidecar" Oct 31 00:50:20.275545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274831 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="662a69f1-ae92-47c8-b1fa-2de72ba91cef" containerName="ta-container" Oct 31 00:50:20.275545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274843 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="config-reloader" Oct 31 00:50:20.275545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274854 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea97dcd-5558-46ff-8ffe-1a86d5cd8345" containerName="thanos-querier" Oct 31 00:50:20.275545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274863 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa6285f-8d00-4c2b-9cfc-db454d834972" containerName="prometheus" Oct 31 00:50:20.275545 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.274873 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ee68b3-d021-4dcc-9e97-c7f764ed986d" containerName="ta-container" Oct 31 00:50:20.280538 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.280515 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:20.286644 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.286511 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:50:20.289521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.289494 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj"] Oct 31 00:50:20.357339 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.357269 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfs57\" (UniqueName: \"kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj\" (UID: \"42cb0cf6-d60f-4907-a3d0-5177655d07e7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:20.458233 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.458201 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfs57\" (UniqueName: \"kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj\" (UID: \"42cb0cf6-d60f-4907-a3d0-5177655d07e7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:20.471252 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.471222 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfs57\" (UniqueName: \"kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj\" (UID: \"42cb0cf6-d60f-4907-a3d0-5177655d07e7\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:20.597941 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.597871 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:20.667963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.667303 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7"] Oct 31 00:50:20.675588 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.675566 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.683733 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.682767 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7"] Oct 31 00:50:20.691483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.691394 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:50:20.691483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.691447 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-n47x7\"" Oct 31 00:50:20.691483 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.691460 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:50:20.760707 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.760679 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.760892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.760714 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.760892 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.760865 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbq27\" (UniqueName: \"kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.861830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.861722 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbq27\" (UniqueName: \"kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.861830 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.861778 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.862047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.861835 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.862472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.862449 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.864484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.864461 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.874578 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.874556 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbq27\" (UniqueName: \"kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27\") pod \"data-science-collector-targetallocator-6c666884cb-85lg7\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:20.889004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.888983 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj"] Oct 31 00:50:20.892268 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:50:20.892245 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42cb0cf6_d60f_4907_a3d0_5177655d07e7.slice/crio-c940237dffc0bf1ca3978f2872ab9cd01ffc999ba3b28f765f73f84a48d7485c WatchSource:0}: Error finding container c940237dffc0bf1ca3978f2872ab9cd01ffc999ba3b28f765f73f84a48d7485c: Status 404 returned error can't find the container with id c940237dffc0bf1ca3978f2872ab9cd01ffc999ba3b28f765f73f84a48d7485c Oct 31 00:50:20.948986 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:20.948959 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:21.000940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.000915 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:21.272956 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.272925 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7"] Oct 31 00:50:21.275616 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:50:21.275591 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31e6356c_ca2e_4366_8d28_34384d220d7b.slice/crio-63b95a6826ff0099959af24eced46c316949cc260b6504c2c97a51adbc25fd4f WatchSource:0}: Error finding container 63b95a6826ff0099959af24eced46c316949cc260b6504c2c97a51adbc25fd4f: Status 404 returned error can't find the container with id 63b95a6826ff0099959af24eced46c316949cc260b6504c2c97a51adbc25fd4f Oct 31 00:50:21.277817 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.277760 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:50:21.277905 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.277844 5073 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 31 00:50:21.559831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.559721 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" event={"ID":"31e6356c-ca2e-4366-8d28-34384d220d7b","Type":"ContainerStarted","Data":"85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9"} Oct 31 00:50:21.559831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.559768 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" event={"ID":"31e6356c-ca2e-4366-8d28-34384d220d7b","Type":"ContainerStarted","Data":"63b95a6826ff0099959af24eced46c316949cc260b6504c2c97a51adbc25fd4f"} Oct 31 00:50:21.559831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.559821 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:21.561431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.561397 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" event={"ID":"42cb0cf6-d60f-4907-a3d0-5177655d07e7","Type":"ContainerStarted","Data":"6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082"} Oct 31 00:50:21.561431 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.561428 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" event={"ID":"42cb0cf6-d60f-4907-a3d0-5177655d07e7","Type":"ContainerStarted","Data":"c940237dffc0bf1ca3978f2872ab9cd01ffc999ba3b28f765f73f84a48d7485c"} Oct 31 00:50:21.561691 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.561678 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:21.580292 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.580251 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" podStartSLOduration=1.580239699 podStartE2EDuration="1.580239699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:21.578632621 +0000 UTC m=+3105.639407313" watchObservedRunningTime="2026-10-31 00:50:21.580239699 +0000 UTC m=+3105.641014394" Oct 31 00:50:21.615914 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:21.615871 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" podStartSLOduration=1.615858635 podStartE2EDuration="1.615858635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:21.614064954 +0000 UTC m=+3105.674839659" watchObservedRunningTime="2026-10-31 00:50:21.615858635 +0000 UTC m=+3105.676633329" Oct 31 00:50:25.792470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:25.792439 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r"] Oct 31 00:50:25.795916 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:25.795893 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:25.805632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:25.805605 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r"] Oct 31 00:50:25.903068 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:25.903041 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsxb\" (UniqueName: \"kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r\" (UID: \"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:26.004506 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.004474 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsxb\" (UniqueName: \"kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r\" (UID: \"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:26.016856 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.016825 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsxb\" (UniqueName: \"kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r\" (UID: \"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:26.109219 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.109159 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:26.384912 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.384833 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r"] Oct 31 00:50:26.387486 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:50:26.387458 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod406ecb0b_6bc8_42dc_9eb4_751a2b91f1b2.slice/crio-873eaf6a72c85f8971adafc634182de723bd07760a5b72c87cd01572cee21c5a WatchSource:0}: Error finding container 873eaf6a72c85f8971adafc634182de723bd07760a5b72c87cd01572cee21c5a: Status 404 returned error can't find the container with id 873eaf6a72c85f8971adafc634182de723bd07760a5b72c87cd01572cee21c5a Oct 31 00:50:26.584686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.584463 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" event={"ID":"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2","Type":"ContainerStarted","Data":"ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d"} Oct 31 00:50:26.584686 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.584524 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" event={"ID":"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2","Type":"ContainerStarted","Data":"873eaf6a72c85f8971adafc634182de723bd07760a5b72c87cd01572cee21c5a"} Oct 31 00:50:26.605813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.605753 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" podStartSLOduration=1.605739556 podStartE2EDuration="1.605739556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:26.604673328 +0000 UTC m=+3110.665448022" watchObservedRunningTime="2026-10-31 00:50:26.605739556 +0000 UTC m=+3110.666514250" Oct 31 00:50:26.632484 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.632453 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj"] Oct 31 00:50:26.632624 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.632606 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" podUID="42cb0cf6-d60f-4907-a3d0-5177655d07e7" containerName="thanos-querier" containerID="cri-o://6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082" gracePeriod=30 Oct 31 00:50:26.826485 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.826456 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:26.831308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.831284 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.839060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.838964 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:50:26.839060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.838976 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:50:26.839060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839033 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-zqm6n\"" Oct 31 00:50:26.839060 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839039 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:50:26.840409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839185 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:50:26.840409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839205 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:50:26.840409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839251 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:50:26.840409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839303 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:50:26.840409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839346 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:50:26.840409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.839610 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:50:26.848026 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.848003 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:26.908945 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.908919 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:26.911063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911041 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911131 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911108 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911181 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911162 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911218 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911201 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911226 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911258 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911249 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911362 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911343 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911391 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4b4\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911450 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911438 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911486 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911469 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911525 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911493 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:26.911564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:26.911540 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.012494 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012468 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57\" (UniqueName: \"kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57\") pod \"42cb0cf6-d60f-4907-a3d0-5177655d07e7\" (UID: \"42cb0cf6-d60f-4907-a3d0-5177655d07e7\") " Oct 31 00:50:27.012636 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012609 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.012690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012637 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.012690 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012667 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.012831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012700 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.012831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012769 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.012831 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012825 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.013047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012855 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.013047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012891 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.013047 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.012925 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.013898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.013677 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.013898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.013739 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4b4\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.013898 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.013769 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.014154 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.013973 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.015001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.014484 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.015001 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.014519 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.015443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.015405 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57" pod "42cb0cf6-d60f-4907-a3d0-5177655d07e7" (UID: "42cb0cf6-d60f-4907-a3d0-5177655d07e7"). InnerVolumeSpecName "kube-api-access-bfs57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:27.016012 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.015979 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.016102 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.016047 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.016198 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.016174 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.016726 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.016703 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.016923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.016900 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.017093 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.017077 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.018074 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.018054 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.018517 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.018477 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.027025 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.027004 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4b4\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:27.115123 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.115091 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfs57\" (UniqueName: \"kubernetes.io/projected/42cb0cf6-d60f-4907-a3d0-5177655d07e7-kube-api-access-bfs57\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:27.590861 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.590818 5073 generic.go:356] "Generic (PLEG): container finished" podID="42cb0cf6-d60f-4907-a3d0-5177655d07e7" containerID="6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082" exitCode=0 Oct 31 00:50:27.591036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.590935 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" Oct 31 00:50:27.591036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.590949 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" event={"ID":"42cb0cf6-d60f-4907-a3d0-5177655d07e7","Type":"ContainerDied","Data":"6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082"} Oct 31 00:50:27.591036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.590995 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj" event={"ID":"42cb0cf6-d60f-4907-a3d0-5177655d07e7","Type":"ContainerDied","Data":"c940237dffc0bf1ca3978f2872ab9cd01ffc999ba3b28f765f73f84a48d7485c"} Oct 31 00:50:27.591036 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.591012 5073 scope.go:122] "RemoveContainer" containerID="6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082" Oct 31 00:50:27.600963 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.600944 5073 scope.go:122] "RemoveContainer" containerID="6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082" Oct 31 00:50:27.601226 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:27.601208 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082\": container with ID starting with 6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082 not found: ID does not exist" containerID="6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082" Oct 31 00:50:27.601284 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.601232 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082"} err="failed to get container status \"6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082\": rpc error: code = NotFound desc = could not find container \"6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082\": container with ID starting with 6c84ac0eca55279b467a20fc4f7fde3365714e70d104f4ff0a750015da524082 not found: ID does not exist" Oct 31 00:50:27.614149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.614112 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj"] Oct 31 00:50:27.620321 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:27.620299 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-9ccpj"] Oct 31 00:50:28.058923 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:28.058895 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cb0cf6-d60f-4907-a3d0-5177655d07e7" path="/var/lib/kubelet/pods/42cb0cf6-d60f-4907-a3d0-5177655d07e7/volumes" Oct 31 00:50:29.232034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.232001 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.239595 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.239569 5073 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.333191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.333161 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.333317 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.333299 5073 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.336880 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.336863 5073 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.428613 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.428574 5073 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.444616 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.444593 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.555191 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.555139 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:29.844650 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:29.844626 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:29.846412 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:50:29.846385 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ae35a8_14f7_45c7_bc5a_27931f8a58d4.slice/crio-3f0ea80ecb973421e57bed44b13514c22b7449046bcf32bb513abd30dc3ad01e WatchSource:0}: Error finding container 3f0ea80ecb973421e57bed44b13514c22b7449046bcf32bb513abd30dc3ad01e: Status 404 returned error can't find the container with id 3f0ea80ecb973421e57bed44b13514c22b7449046bcf32bb513abd30dc3ad01e Oct 31 00:50:30.607012 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:30.606977 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerStarted","Data":"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c"} Oct 31 00:50:30.607478 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:30.607043 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerStarted","Data":"3f0ea80ecb973421e57bed44b13514c22b7449046bcf32bb513abd30dc3ad01e"} Oct 31 00:50:30.949520 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:30.949444 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:33.623366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:33.623290 5073 generic.go:356] "Generic (PLEG): container finished" podID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerID="aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c" exitCode=0 Oct 31 00:50:33.623759 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:33.623361 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerDied","Data":"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c"} Oct 31 00:50:34.630308 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:34.630273 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerStarted","Data":"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb"} Oct 31 00:50:34.630700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:34.630340 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerStarted","Data":"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37"} Oct 31 00:50:34.630700 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:34.630363 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerStarted","Data":"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21"} Oct 31 00:50:34.663051 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:34.662995 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=5.578412309 podStartE2EDuration="8.66297638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.084564071s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:34.661609764 +0000 UTC m=+3118.722384460" watchObservedRunningTime="2026-10-31 00:50:34.66297638 +0000 UTC m=+3118.723751077" Oct 31 00:50:39.555775 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:39.555737 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:40.949604 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:40.949560 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:43.085392 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:43.085353 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:50:44.556283 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:44.556256 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:44.561424 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:44.561404 5073 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:44.679318 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:44.679295 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:50.948652 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:50.948612 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:53.859500 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:53.859463 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r"] Oct 31 00:50:53.860277 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:53.859718 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" podUID="406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" containerName="thanos-querier" containerID="cri-o://ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d" gracePeriod=30 Oct 31 00:50:53.946798 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:53.946741 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:53.947212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:53.947024 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="prometheus" containerID="cri-o://9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" gracePeriod=600 Oct 31 00:50:53.947212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:53.947076 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="thanos-sidecar" containerID="cri-o://b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" gracePeriod=600 Oct 31 00:50:53.947212 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:53.947081 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="config-reloader" containerID="cri-o://02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" gracePeriod=600 Oct 31 00:50:54.156462 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.156431 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7"] Oct 31 00:50:54.156632 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.156607 5073 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" podUID="31e6356c-ca2e-4366-8d28-34384d220d7b" containerName="ta-container" containerID="cri-o://85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9" gracePeriod=30 Oct 31 00:50:54.394409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.394101 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:54.419841 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.419220 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb\" (UniqueName: \"kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb\") pod \"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2\" (UID: \"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2\") " Oct 31 00:50:54.422324 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.422296 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb" pod "406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" (UID: "406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2"). InnerVolumeSpecName "kube-api-access-jtsxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.520224 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.520187 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtsxb\" (UniqueName: \"kubernetes.io/projected/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2-kube-api-access-jtsxb\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.534463 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.534434 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:54.621958 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.621931 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal\") pod \"31e6356c-ca2e-4366-8d28-34384d220d7b\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " Oct 31 00:50:54.622104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.622021 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27\" (UniqueName: \"kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27\") pod \"31e6356c-ca2e-4366-8d28-34384d220d7b\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " Oct 31 00:50:54.622104 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.622050 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert\") pod \"31e6356c-ca2e-4366-8d28-34384d220d7b\" (UID: \"31e6356c-ca2e-4366-8d28-34384d220d7b\") " Oct 31 00:50:54.622338 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.622313 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal" pod "31e6356c-ca2e-4366-8d28-34384d220d7b" (UID: "31e6356c-ca2e-4366-8d28-34384d220d7b"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.624992 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.624970 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert" pod "31e6356c-ca2e-4366-8d28-34384d220d7b" (UID: "31e6356c-ca2e-4366-8d28-34384d220d7b"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.625386 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.625366 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27" pod "31e6356c-ca2e-4366-8d28-34384d220d7b" (UID: "31e6356c-ca2e-4366-8d28-34384d220d7b"). InnerVolumeSpecName "kube-api-access-pbq27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.722564 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.722533 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:54.723091 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.723065 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbq27\" (UniqueName: \"kubernetes.io/projected/31e6356c-ca2e-4366-8d28-34384d220d7b-kube-api-access-pbq27\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.723235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.723093 5073 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/31e6356c-ca2e-4366-8d28-34384d220d7b-data-science-collector-ta-server-cert\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.723235 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.723109 5073 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/31e6356c-ca2e-4366-8d28-34384d220d7b-ta-internal\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.725242 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.725219 5073 generic.go:356] "Generic (PLEG): container finished" podID="406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" containerID="ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d" exitCode=0 Oct 31 00:50:54.725363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.725265 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" Oct 31 00:50:54.725363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.725290 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" event={"ID":"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2","Type":"ContainerDied","Data":"ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d"} Oct 31 00:50:54.725363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.725323 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r" event={"ID":"406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2","Type":"ContainerDied","Data":"873eaf6a72c85f8971adafc634182de723bd07760a5b72c87cd01572cee21c5a"} Oct 31 00:50:54.725363 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.725340 5073 scope.go:122] "RemoveContainer" containerID="ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d" Oct 31 00:50:54.729261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.729069 5073 generic.go:356] "Generic (PLEG): container finished" podID="31e6356c-ca2e-4366-8d28-34384d220d7b" containerID="85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9" exitCode=0 Oct 31 00:50:54.729261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.729110 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" Oct 31 00:50:54.729261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.729194 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" event={"ID":"31e6356c-ca2e-4366-8d28-34384d220d7b","Type":"ContainerDied","Data":"85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9"} Oct 31 00:50:54.729261 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.729226 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7" event={"ID":"31e6356c-ca2e-4366-8d28-34384d220d7b","Type":"ContainerDied","Data":"63b95a6826ff0099959af24eced46c316949cc260b6504c2c97a51adbc25fd4f"} Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736311 5073 generic.go:356] "Generic (PLEG): container finished" podID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerID="b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" exitCode=0 Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736333 5073 generic.go:356] "Generic (PLEG): container finished" podID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerID="02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" exitCode=0 Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736341 5073 generic.go:356] "Generic (PLEG): container finished" podID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerID="9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" exitCode=0 Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736390 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerDied","Data":"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb"} Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736402 5073 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736447 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerDied","Data":"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37"} Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736489 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerDied","Data":"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21"} Oct 31 00:50:54.736623 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.736526 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"66ae35a8-14f7-45c7-bc5a-27931f8a58d4","Type":"ContainerDied","Data":"3f0ea80ecb973421e57bed44b13514c22b7449046bcf32bb513abd30dc3ad01e"} Oct 31 00:50:54.737307 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.737286 5073 scope.go:122] "RemoveContainer" containerID="ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d" Oct 31 00:50:54.737582 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:54.737558 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d\": container with ID starting with ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d not found: ID does not exist" containerID="ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d" Oct 31 00:50:54.737657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.737589 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d"} err="failed to get container status \"ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d\": rpc error: code = NotFound desc = could not find container \"ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d\": container with ID starting with ba11f1c498094f96663508f0365e1d75d2979c5e13c7cdfc304d4121e730cd7d not found: ID does not exist" Oct 31 00:50:54.737657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.737605 5073 scope.go:122] "RemoveContainer" containerID="85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9" Oct 31 00:50:54.745377 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.745359 5073 scope.go:122] "RemoveContainer" containerID="85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9" Oct 31 00:50:54.745626 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:54.745609 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9\": container with ID starting with 85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9 not found: ID does not exist" containerID="85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9" Oct 31 00:50:54.745671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.745633 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9"} err="failed to get container status \"85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9\": rpc error: code = NotFound desc = could not find container \"85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9\": container with ID starting with 85e664c2858dcde589b1d95b108427d21f1c17ba4707b32b51d4599a92e3ecb9 not found: ID does not exist" Oct 31 00:50:54.745671 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.745647 5073 scope.go:122] "RemoveContainer" containerID="b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" Oct 31 00:50:54.754521 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.754502 5073 scope.go:122] "RemoveContainer" containerID="02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" Oct 31 00:50:54.765813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.765776 5073 scope.go:122] "RemoveContainer" containerID="9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" Oct 31 00:50:54.770155 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.770132 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7"] Oct 31 00:50:54.774193 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.774173 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-6c666884cb-85lg7"] Oct 31 00:50:54.776885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.776723 5073 scope.go:122] "RemoveContainer" containerID="aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c" Oct 31 00:50:54.786493 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.786475 5073 scope.go:122] "RemoveContainer" containerID="b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" Oct 31 00:50:54.786772 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:54.786754 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": container with ID starting with b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb not found: ID does not exist" containerID="b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" Oct 31 00:50:54.786885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.786804 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb"} err="failed to get container status \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": rpc error: code = NotFound desc = could not find container \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": container with ID starting with b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb not found: ID does not exist" Oct 31 00:50:54.786885 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.786823 5073 scope.go:122] "RemoveContainer" containerID="02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" Oct 31 00:50:54.787086 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:54.787066 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": container with ID starting with 02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37 not found: ID does not exist" containerID="02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" Oct 31 00:50:54.787152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.787094 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37"} err="failed to get container status \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": rpc error: code = NotFound desc = could not find container \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": container with ID starting with 02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37 not found: ID does not exist" Oct 31 00:50:54.787152 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.787115 5073 scope.go:122] "RemoveContainer" containerID="9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" Oct 31 00:50:54.787422 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:54.787405 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": container with ID starting with 9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21 not found: ID does not exist" containerID="9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" Oct 31 00:50:54.787488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.787426 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21"} err="failed to get container status \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": rpc error: code = NotFound desc = could not find container \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": container with ID starting with 9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21 not found: ID does not exist" Oct 31 00:50:54.787488 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.787441 5073 scope.go:122] "RemoveContainer" containerID="aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c" Oct 31 00:50:54.787687 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:50:54.787668 5073 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": container with ID starting with aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c not found: ID does not exist" containerID="aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c" Oct 31 00:50:54.787774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.787694 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c"} err="failed to get container status \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": rpc error: code = NotFound desc = could not find container \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": container with ID starting with aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c not found: ID does not exist" Oct 31 00:50:54.787774 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.787714 5073 scope.go:122] "RemoveContainer" containerID="b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.788034 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb"} err="failed to get container status \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": rpc error: code = NotFound desc = could not find container \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": container with ID starting with b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.788060 5073 scope.go:122] "RemoveContainer" containerID="02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.788299 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37"} err="failed to get container status \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": rpc error: code = NotFound desc = could not find container \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": container with ID starting with 02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37 not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.788320 5073 scope.go:122] "RemoveContainer" containerID="9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.788600 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21"} err="failed to get container status \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": rpc error: code = NotFound desc = could not find container \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": container with ID starting with 9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21 not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.788628 5073 scope.go:122] "RemoveContainer" containerID="aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.790773 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r"] Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.790977 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c"} err="failed to get container status \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": rpc error: code = NotFound desc = could not find container \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": container with ID starting with aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.790997 5073 scope.go:122] "RemoveContainer" containerID="b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791210 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb"} err="failed to get container status \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": rpc error: code = NotFound desc = could not find container \"b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb\": container with ID starting with b33112a84676ddad5057b5cf923802951d5d603a72568366f1a88864ceed65bb not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791227 5073 scope.go:122] "RemoveContainer" containerID="02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791437 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37"} err="failed to get container status \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": rpc error: code = NotFound desc = could not find container \"02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37\": container with ID starting with 02a969f1a02676978a7a39d09e18252da9e40997e83a19ebf75ac151d170df37 not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791448 5073 scope.go:122] "RemoveContainer" containerID="9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791648 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21"} err="failed to get container status \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": rpc error: code = NotFound desc = could not find container \"9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21\": container with ID starting with 9118c1224e4453696984e1e7c5b7819a9c58050d64ba877b498767f34136ce21 not found: ID does not exist" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791665 5073 scope.go:122] "RemoveContainer" containerID="aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c" Oct 31 00:50:54.792063 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.791991 5073 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c"} err="failed to get container status \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": rpc error: code = NotFound desc = could not find container \"aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c\": container with ID starting with aea7e32878d9b654d7baffd15972260d5dc16e836c21dbf029ff6364c0ca749c not found: ID does not exist" Oct 31 00:50:54.794902 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.794881 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-7m88r"] Oct 31 00:50:54.823657 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823637 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.823751 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823669 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.823834 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823813 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.823890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823871 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.823948 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823910 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823963 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824013 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.823995 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824047 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824085 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824138 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824116 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824154 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824310 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824207 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.824529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824340 5073 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config\") pod \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\" (UID: \"66ae35a8-14f7-45c7-bc5a-27931f8a58d4\") " Oct 31 00:50:54.825199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.824886 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.825199 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.825159 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.825366 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.825218 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.826512 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.826485 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "kube-api-access-nw4b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.826891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.826844 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.827164 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.827140 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.828940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.828914 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.828940 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.828931 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.829139 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.828917 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.829305 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.829278 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.829997 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.829977 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:50:54.830470 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.830448 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.839890 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.839872 5073 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config" pod "66ae35a8-14f7-45c7-bc5a-27931f8a58d4" (UID: "66ae35a8-14f7-45c7-bc5a-27931f8a58d4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.925472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925444 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925472 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925469 5073 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-tls-assets\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925481 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925493 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925503 5073 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925511 5073 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925519 5073 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4b4\" (UniqueName: \"kubernetes.io/projected/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-kube-api-access-nw4b4\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925528 5073 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925552 5073 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" " Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925562 5073 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-secret-prometheus-web-tls-ca\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925571 5073 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925580 5073 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.925848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.925590 5073 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66ae35a8-14f7-45c7-bc5a-27931f8a58d4-config-out\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.936685 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:54.936669 5073 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-140-246.us-east-2.compute.internal" Oct 31 00:50:55.026095 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:55.026073 5073 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-140-246.us-east-2.compute.internal\" DevicePath \"csi-5007b703de5ca06bc7069c373d67917e5424cae80e7d5016c13d305f9cc103e2\"" Oct 31 00:50:55.060524 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:55.060503 5073 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:55.067697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:55.067674 5073 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:50:56.058569 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:56.058537 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e6356c-ca2e-4366-8d28-34384d220d7b" path="/var/lib/kubelet/pods/31e6356c-ca2e-4366-8d28-34384d220d7b/volumes" Oct 31 00:50:56.058974 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:56.058854 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" path="/var/lib/kubelet/pods/406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2/volumes" Oct 31 00:50:56.059247 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:56.059234 5073 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" path="/var/lib/kubelet/pods/66ae35a8-14f7-45c7-bc5a-27931f8a58d4/volumes" Oct 31 00:50:57.675951 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:50:57.675908 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="prometheus" probeResult="failure" output="Get \"https://10.132.0.222:9090/-/ready\": context deadline exceeded" Oct 31 00:51:00.948670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:00.948625 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:10.949410 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:10.949371 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:13.084650 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:51:13.084615 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:51:20.949449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:20.949401 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:24.257376 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:24.257339 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qkj2v_106281ea-1622-4966-bfd3-bdba5d770f5f/global-pull-secret-syncer/0.log" Oct 31 00:51:24.367358 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:24.367333 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-52d97_93866d6e-e48c-4e7f-823d-771497d4694b/connection-checker/0.log" Oct 31 00:51:24.399108 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:24.399082 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tz88x_3da2a0da-b9b7-41f0-8e8d-6e07faa06d99/connection-checker/0.log" Oct 31 00:51:24.431449 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:24.431432 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-z49dz_3ed58e8f-b3ed-4c94-aebf-0676c621799d/connection-checker/0.log" Oct 31 00:51:24.548146 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:24.548089 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v79mw_86e12f20-75a3-480a-81b1-f6bf080a6733/konnectivity-agent/0.log" Oct 31 00:51:24.724770 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:24.724748 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-246.us-east-2.compute.internal_fc47763c26c267f0abea2ce914234e24/haproxy/0.log" Oct 31 00:51:30.571612 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:30.571564 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-whqjq_7dbbbfcf-c966-49b9-8cf3-91b77469bd2b/cluster-monitoring-operator/0.log" Oct 31 00:51:30.949553 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:30.949472 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:31.042735 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:31.042712 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nh72c_08c22c12-34b8-4338-9558-81ad92772a96/node-exporter/0.log" Oct 31 00:51:31.074758 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:31.074741 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nh72c_08c22c12-34b8-4338-9558-81ad92772a96/kube-rbac-proxy/0.log" Oct 31 00:51:31.124829 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:31.124810 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nh72c_08c22c12-34b8-4338-9558-81ad92772a96/init-textfile/0.log" Oct 31 00:51:31.534465 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:31.534437 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2ng7b_e1d53cc2-f01d-4b48-91b3-23717c206abd/prometheus-operator/0.log" Oct 31 00:51:31.556938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:31.556919 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2ng7b_e1d53cc2-f01d-4b48-91b3-23717c206abd/kube-rbac-proxy/0.log" Oct 31 00:51:31.591443 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:31.591418 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qgqzr_cef65b43-4a0c-4b34-a4e9-579f817084e1/prometheus-operator-admission-webhook/0.log" Oct 31 00:51:32.957765 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.957735 5073 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf"] Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958106 5073 cpu_manager.go:414] "containerMap: removing container" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="thanos-sidecar" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958119 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="thanos-sidecar" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958134 5073 cpu_manager.go:414] "containerMap: removing container" podUID="42cb0cf6-d60f-4907-a3d0-5177655d07e7" containerName="thanos-querier" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958139 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cb0cf6-d60f-4907-a3d0-5177655d07e7" containerName="thanos-querier" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958152 5073 cpu_manager.go:414] "containerMap: removing container" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="init-config-reloader" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958157 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="init-config-reloader" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958166 5073 cpu_manager.go:414] "containerMap: removing container" podUID="31e6356c-ca2e-4366-8d28-34384d220d7b" containerName="ta-container" Oct 31 00:51:32.958167 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958171 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e6356c-ca2e-4366-8d28-34384d220d7b" containerName="ta-container" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958180 5073 cpu_manager.go:414] "containerMap: removing container" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="prometheus" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958185 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="prometheus" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958194 5073 cpu_manager.go:414] "containerMap: removing container" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="config-reloader" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958198 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="config-reloader" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958208 5073 cpu_manager.go:414] "containerMap: removing container" podUID="406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" containerName="thanos-querier" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958212 5073 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" containerName="thanos-querier" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958270 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cb0cf6-d60f-4907-a3d0-5177655d07e7" containerName="thanos-querier" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958279 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e6356c-ca2e-4366-8d28-34384d220d7b" containerName="ta-container" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958286 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="prometheus" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958292 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="config-reloader" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958297 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="406ecb0b-6bc8-42dc-9eb4-751a2b91f1b2" containerName="thanos-querier" Oct 31 00:51:32.958446 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.958304 5073 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ae35a8-14f7-45c7-bc5a-27931f8a58d4" containerName="thanos-sidecar" Oct 31 00:51:32.961409 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.961390 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:32.971350 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.971326 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf"] Oct 31 00:51:32.971813 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.971778 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtkzz\"/\"openshift-service-ca.crt\"" Oct 31 00:51:32.971874 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.971868 5073 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xtkzz\"/\"default-dockercfg-bmqgh\"" Oct 31 00:51:32.971934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:32.971924 5073 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtkzz\"/\"kube-root-ca.crt\"" Oct 31 00:51:33.004692 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.004626 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6d8\" (UniqueName: \"kubernetes.io/projected/109edd06-c5a1-44cc-a086-f13e3b230937-kube-api-access-bc6d8\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.004692 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.004674 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-proc\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.004891 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.004825 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-lib-modules\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.004934 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.004893 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-sys\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.004973 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.004942 5073 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-podres\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105427 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105400 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-sys\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105445 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-podres\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105495 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6d8\" (UniqueName: \"kubernetes.io/projected/109edd06-c5a1-44cc-a086-f13e3b230937-kube-api-access-bc6d8\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105540 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105527 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-proc\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105536 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-sys\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105545 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-podres\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105633 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-proc\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.105748 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105646 5073 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-lib-modules\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.106035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.105747 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/109edd06-c5a1-44cc-a086-f13e3b230937-lib-modules\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.118676 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.118650 5073 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6d8\" (UniqueName: \"kubernetes.io/projected/109edd06-c5a1-44cc-a086-f13e3b230937-kube-api-access-bc6d8\") pod \"perf-node-gather-daemonset-pjvmf\" (UID: \"109edd06-c5a1-44cc-a086-f13e3b230937\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.273382 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.273320 5073 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.326004 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.325975 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-46klc_b2f8fec0-412d-40ee-9d94-db3fbe04a651/networking-console-plugin/0.log" Oct 31 00:51:33.547044 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.547016 5073 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf"] Oct 31 00:51:33.549604 ip-10-0-140-246 kubenswrapper[5073]: W1031 00:51:33.549573 5073 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod109edd06_c5a1_44cc_a086_f13e3b230937.slice/crio-bd682e29756deb1ed7a3da3e7e832e740cc1f7d8356c3ad6adbac80b78308314 WatchSource:0}: Error finding container bd682e29756deb1ed7a3da3e7e832e740cc1f7d8356c3ad6adbac80b78308314: Status 404 returned error can't find the container with id bd682e29756deb1ed7a3da3e7e832e740cc1f7d8356c3ad6adbac80b78308314 Oct 31 00:51:33.906529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.906463 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" event={"ID":"109edd06-c5a1-44cc-a086-f13e3b230937","Type":"ContainerStarted","Data":"0da5006f8c472886a2f5dfd53d52a6101376fa06ca53646c543eabde16409b64"} Oct 31 00:51:33.906529 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.906514 5073 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" event={"ID":"109edd06-c5a1-44cc-a086-f13e3b230937","Type":"ContainerStarted","Data":"bd682e29756deb1ed7a3da3e7e832e740cc1f7d8356c3ad6adbac80b78308314"} Oct 31 00:51:33.906703 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.906606 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:33.924346 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:33.924299 5073 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" podStartSLOduration=1.9242844529999998 podStartE2EDuration="1.924284453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:33.923762883 +0000 UTC m=+3177.984537964" watchObservedRunningTime="2026-10-31 00:51:33.924284453 +0000 UTC m=+3177.985059147" Oct 31 00:51:34.029057 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:34.029019 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/1.log" Oct 31 00:51:34.032583 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:34.032563 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sm45j_5fa50726-3415-4c7f-82ba-e6eb5115267a/console-operator/2.log" Oct 31 00:51:34.568399 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:34.568362 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bdd949fc8-ct86f_0dd4a18f-ba45-441f-9f8a-e2fccc32b77c/console/0.log" Oct 31 00:51:34.605034 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:34.605010 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-5vs22_3ffa894e-03f3-4791-9afb-76d99ec77a24/download-server/0.log" Oct 31 00:51:35.216633 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:35.216600 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-px2hc_b470b4c4-501f-4d51-a04d-37f1f3d71c06/volume-data-source-validator/0.log" Oct 31 00:51:36.256848 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:36.256804 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bnkm_e1a27ea3-18d4-4aff-a92b-a4290ec72714/dns/0.log" Oct 31 00:51:36.296532 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:36.296498 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bnkm_e1a27ea3-18d4-4aff-a92b-a4290ec72714/kube-rbac-proxy/0.log" Oct 31 00:51:36.456719 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:36.456698 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjnpt_b613d66f-e947-427e-9694-3f0b609f35c4/dns-node-resolver/0.log" Oct 31 00:51:37.078392 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:37.078344 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dd666768c-vdtlt_e412acfd-c25b-4802-8d58-d99dd3390098/registry/0.log" Oct 31 00:51:37.126864 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:37.126839 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n28vw_71c479d0-cb02-4549-acb7-3a206eb2ecb7/node-ca/0.log" Oct 31 00:51:38.002098 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:38.002074 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f8b974b69-pvb5d_6df4d978-5bb3-48e6-a69f-519f0d85e465/router/0.log" Oct 31 00:51:38.420697 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:38.420604 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nl6l6_f270b734-9691-480f-8d55-1448c307a69b/serve-healthcheck-canary/0.log" Oct 31 00:51:38.813670 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:38.813644 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-nnh9c_beff8893-2560-45f8-a96d-bce1e51dcf80/insights-operator/0.log" Oct 31 00:51:38.814341 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:38.814322 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-nnh9c_beff8893-2560-45f8-a96d-bce1e51dcf80/insights-operator/1.log" Oct 31 00:51:38.979932 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:38.979905 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvn2x_aeeea8e9-d19d-4518-b9cb-285b96d9a5d1/kube-rbac-proxy/0.log" Oct 31 00:51:39.003079 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:39.003060 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvn2x_aeeea8e9-d19d-4518-b9cb-285b96d9a5d1/exporter/0.log" Oct 31 00:51:39.030481 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:39.030459 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvn2x_aeeea8e9-d19d-4518-b9cb-285b96d9a5d1/extractor/0.log" Oct 31 00:51:39.063327 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:39.063305 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-bbltk-htq8x_d91a42d7-9685-4f8f-b620-5e93d58b1a76/insights-gathering/0.log" Oct 31 00:51:39.092574 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:39.092530 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-bhj77-m945q_bf3d2e09-a636-401f-9852-3a74004cfc1b/insights-gathering/0.log" Oct 31 00:51:39.921170 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:39.921144 5073 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-pjvmf" Oct 31 00:51:40.417035 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:40.417003 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-operators-redhat_loki-operator-controller-manager-5666dc7fd5-lfnkl_c8f643e2-9c77-4eb6-a0bd-bc9c99033732/manager/0.log" Oct 31 00:51:40.949666 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:40.949634 5073 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-0" podUID="c57125c0-82f4-40e5-b8ea-aeba9fdab92c" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:43.084610 ip-10-0-140-246 kubenswrapper[5073]: E1031 00:51:43.084578 5073 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jvn2x" containerName="extractor" Oct 31 00:51:46.584149 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:46.584053 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jwqps_686b1996-b6f2-44e9-b9bd-8acb156e833d/migrator/0.log" Oct 31 00:51:46.617419 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:46.617380 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wmdc4_d0ffac65-ad93-4617-afbd-9d2fe6153391/migrator/0.log" Oct 31 00:51:46.976996 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:46.976930 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc_6afa2442-496b-45d4-ae52-133e0d21a60f/kube-storage-version-migrator-operator/1.log" Oct 31 00:51:46.977886 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:46.977868 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jtpcc_6afa2442-496b-45d4-ae52-133e0d21a60f/kube-storage-version-migrator-operator/0.log" Oct 31 00:51:48.068929 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.068898 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/kube-multus-additional-cni-plugins/0.log" Oct 31 00:51:48.093877 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.093850 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/egress-router-binary-copy/0.log" Oct 31 00:51:48.117675 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.117657 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/cni-plugins/0.log" Oct 31 00:51:48.144993 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.144973 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/bond-cni-plugin/0.log" Oct 31 00:51:48.169111 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.169086 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/routeoverride-cni/0.log" Oct 31 00:51:48.192713 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.192680 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/whereabouts-cni-bincopy/0.log" Oct 31 00:51:48.216715 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.216689 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fpcf_fbea15b0-87b2-480b-9bb7-496f71990632/whereabouts-cni/0.log" Oct 31 00:51:48.762767 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.762733 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wk7mn_8a607b81-bbfa-4a4e-9828-2065d1e3bf49/kube-multus/0.log" Oct 31 00:51:48.909938 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.909911 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dzqgd_96ddf74c-7e70-4e9d-b5fd-77203f8d2179/network-metrics-daemon/0.log" Oct 31 00:51:48.929180 ip-10-0-140-246 kubenswrapper[5073]: I1031 00:51:48.929149 5073 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dzqgd_96ddf74c-7e70-4e9d-b5fd-77203f8d2179/kube-rbac-proxy/0.log"